status.ready is false despite 3scale actually running

[david-martin]

oc get threescale/3scale -n 3scale -o yaml | grep -A5 status
status:
  phase: reconcile
  ready: false
  version: 2.2.0.GA

oc get po -n 3scale
NAME                               READY     STATUS    RESTARTS   AGE
3scale-operator-5cbc5c4887-x6x8z   1/1       Running   0          7d
apicast-production-1-xgzzk         1/1       Running   5          7d
apicast-staging-1-7hsvl            1/1       Running   0          7d
apicast-wildcard-router-1-j62bf    1/1       Running   0          7d
backend-cron-1-p7rh9               1/1       Running   0          7d
backend-listener-1-kgcjt           1/1       Running   0          7d
backend-redis-1-6kpxm              1/1       Running   0          7d
backend-worker-1-mdm5b             1/1       Running   0          7d
system-app-1-hm4lr                 3/3       Running   0          7d
system-memcache-1-7nkts            1/1       Running   0          7d
system-mysql-1-dt858               1/1       Running   0          7d
system-redis-1-vglft               1/1       Running   0          7d
system-resque-1-m4dr9              2/2       Running   0          7d
system-sidekiq-1-7bf44             1/1       Running   1          7d
system-sphinx-1-pj5jb              1/1       Running   2          7d
zync-1-wf9mr                       1/1       Running   1          7d
zync-database-1-r7vd2              1/1       Running   0          7d

This is causing failures in integr8ly/installation where it's using this cmd in a loop to check if 3scale is ready

fatal: [127.0.0.1]: FAILED! => {"attempts": 50, "changed": false, "cmd": "sleep 5; oc get threescale/3scale -o template --template \\{\\{.status.ready\\}\\} --namespace 3scale | grep true", "delta": "0:00:05.146579", "end": "2018-11-14 16:46:08.099148", "failed": true, "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-11-14 16:46:02.952569", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[david-martin]

This is repeatedly output to the operator log

time="2018-11-15T07:45:58Z" level=info msg=ReconcileThreeScale
  | time="2018-11-15T07:46:03Z" level=info msg="Create Resources: done"
  | time="2018-11-15T07:46:03Z" level=info msg="Resources Ready: yes"
  | time="2018-11-15T07:46:03Z" level=error msg="error syncing key (3scale/3scale): phase reconcile threescale failed: error reconciling auth providers: error retrieving auth providers from threescale: failed to LIST authentication_providers: (403) 403 Forbidden"

[david-martin]

@mikenairn Note that I would have run the integreatly installation playbook a number of times against this cluster. Is it possible there's something in the 3scale install that is not idempotent?

[mikenairn]

@david-martin How would you go about re-creating this? Do you have a cluster anywhere that is still in this sate?

[david-martin]

@mikenairn I don't. It was in PDS, but now torn down.

To reproduce, the best guess I have is to install fresh, then re-run the install playbooks to see if the re-run changes something. I'll try reproduce if I get some time