search

ansible / awx-operator

An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖
https://www.github.com/ansible/awx
Apache License 2.0
1.25k stars 628 forks source link

No migration from operator to database #757

Closed larsrei closed 2 years ago

larsrei commented 2 years ago
ISSUE TYPE
SUMMARY

Install awx over awx-operator. For database we use external postgres. From the pod the postgres is reachable and I can login with user and passwor to the database. But no migrations from awx take place. Operator and pods are running. No access to web gui because database not ready.

 kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-6567f7556f-74lqx   2/2     Running   0          6h2m
awx-trainroom-test-844b757bc9-47fxt                4/4     Running   54         6h
ENVIRONMENT
AWX-OPERATOR LOGS
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-ee DEBUG 2022/01/14 20:54:35 Sending service advertisement: &{awx-trainroom-test-844b757bc9-47fxt control 2022-01-14 20:54:35.186449491 +0000 UTC m=+20885.789600748 1 map[type:Control Service] [{local false} {kubernetes-runtime-auth false} {kubernetes-incluster-auth false}]}
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Attempt 12 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Attempt 12 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Attempt 13 of 30
unexpected error: stream error: stream ID 131; INTERNAL_ERROR
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Attempt 13 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-ee DEBUG 2022/01/14 20:55:35 Sending service advertisement: &{awx-trainroom-test-844b757bc9-47fxt control 2022-01-14 20:55:35.186605905 +0000 UTC m=+20945.789757262 1 map[type:Control Service] [{local false} {kubernetes-runtime-auth false} {kubernetes-incluster-auth false}]}
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Attempt 14 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Attempt 14 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-ee DEBUG 2022/01/14 20:56:35 Sending service advertisement: &{awx-trainroom-test-844b757bc9-47fxt control 2022-01-14 20:56:35.186669378 +0000 UTC m=+21005.789820735 1 map[type:Control Service] [{local false} {kubernetes-runtime-auth false} {kubernetes-incluster-auth false}]}
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Attempt 15 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Attempt 15 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Attempt 16 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Attempt 16 of 30
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-task [wait-for-migrations] Waiting 30 seconds before next attempt
awx-trainroom-test-844b757bc9-47fxt awx-trainroom-test-web [wait-for-migrations] Waiting 30 seconds before next attempt

https://mycase.cloudapps.cisco.com/temporaryUpload?requestID=b2R89Z1tP7O13l7R%2FMayfOdackAhZATQ3yrfrlJgpNH%2F8qY7TC0R7jgz7%2B4aP1q6Q6r5qOvSkQbqIDwYeGBLPUGB7BjcuCWbZZvP6Zid8AHbBE7Iavi0wn9JWG7%2FS3HX9ualW0Kl9tdu83iZyK8tleU9Dn%2FOnGmAkhTppoNUrWT4o4KcRgPPJIqb1qZ1FXs2awx-operator-controller-manager-6567f7556f-74lqx awx-manager {"level":"error","ts":1642193593.662564,"logger":"logging_event_handler","msg":"","name":"awx-trainroom-test","namespace":"trainrooms-dev","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"667244732726012636","EventData.Task":"Check for pending migrations","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/installer/tasks/main.yml:51","error":"[playbook task failed]"}
awx-operator-controller-manager-6567f7556f-74lqx awx-manager
awx-operator-controller-manager-6567f7556f-74lqx awx-manager --------------------------- Ansible Task StdOut -------------------------------
awx-operator-controller-manager-6567f7556f-74lqx awx-manager
awx-operator-controller-manager-6567f7556f-74lqx awx-manager  TASK [Check for pending migrations] ********************************
awx-operator-controller-manager-6567f7556f-74lqx awx-manager fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to execute on pod awx-trainroom-test-844b757bc9-47fxt due to : (0)\nReason: Handshake status 500 Internal Server Error\n"}
awx-operator-controller-manager-6567f7556f-74lqx awx-manager
awx-operator-controller-manager-6567f7556f-74lqx awx-manager -------------------------------------------------------------------------------
awx-operator-controller-manager-6567f7556f-74lqx awx-manager {"level":"error","ts":1642193593.85498,"logger":"runner","msg":"ansible-playbook 2.9.19\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\r\n  executable location = /usr/local/b
.....
ks/main.yml:51\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed to execute on pod awx-trainroom-test-844b757bc9-47fxt due to : (0)\\nReason: Handshake status 500 Internal Server Error\\n\"}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=41   changed=0    unreachable=0    failed=1    skipped=35   rescued=0    ignored=0   \r\n\n","job":"667244732726012636","name":"awx-trainroom-test","namespace":"trainrooms-dev","error":"exit status 2"}
shanemcd commented 2 years ago

Hello,

Based on the logs unexpected error: stream error: stream ID 131; INTERNAL_ERROR and from a quick perusal on Google, the issue seems to be with your Kubernetes cluster and not a problem in AWX.

If you need help troubleshooting or are looking for help using AWX, try our mailing list or IRC channel:

ansible-awx on https://libera.chat/

https://groups.google.com/forum/#!forum/awx-project

TheKangaroo commented 2 years ago

Hi @shanemcd. I'm pretty sure the

unexpected error: stream error: stream ID 131; INTERNAL_ERROR

is from stern (https://github.com/wercker/stern) log output and unrelated to the actual issue.