craph
commented
6 months ago This issue looks similar to https://github.com/ansible/awx-operator/issues/1775 and https://github.com/ansible/awx-operator/issues/1770
Open dark-vex opened 6 months ago
craph
commented
6 months ago This issue looks similar to https://github.com/ansible/awx-operator/issues/1775 and https://github.com/ansible/awx-operator/issues/1770
fosterseth
commented
6 months ago can you test my patch here? https://github.com/ansible/awx-operator/issues/1770#issuecomment-2000166172 thanks!
dark-vex
commented
6 months ago @fosterseth thanks, I'll try in another environment. On this one, I've ended up in reinstall+restore data from a backup
LukWe99
commented
6 months ago I encountered a similar problem: In my case the awx-task pods were stuck in init-container "init-database" with "waiting for migrations". Therefore I checked the logs of the awx-operator and I found the following error:
TASK [Verify the resource pod name is populated.] ******************************** fatal: [localhost]: FAILED! => { "assertion": "awx_web_pod_name != ''", "changed": false, "evaluated_to": false, "msg": "Could not find the tower pod's name." }
After checking the awx-operator source code, I think, that removing the "wait" and "wait_timeout" from the task, where the web and task deployments are applied ("Apply deployment resources" in resources_configuration.yml), may cause the problem (Commit ffba1b4, Pull Request #1674).
The deployments are applied without waiting for them to be running. In the immediately following task "Get the new resource pod information after updating resource" the playbook tries to get the infos from the web pods but only with "status.phase=Running". As the previous task is not waiting for the pods created by the deployments to be running, the registered _new_pod variable may be empty at this moment. Therefore all the following set_fact tasks may use empty values and therefore the assertion task "Verify the resource pod name is populated" is failing. The playbook then ends at this point and all the following includes like "migrate_schema.yml", "initialize_django.yml" etc. are not executed.
Please confirm the following
Bug Summary
Hello 👋 , I have upgraded AWX Operator to 2.13.1 using the helm chart but awx-task pod is stuck in "Waiting for database migrations..." phase.
AWX Operator version
2.13.1
AWX version
24.0.0
Kubernetes platform
kubernetes
Kubernetes/Platform version
v1.27.8+k3s2
Modifications
no
Steps to reproduce
I don't have specific steps to reproduce, I have only upgraded from AWX 23.9.0 to 24.0.0 using the helm chart
Expected results
Migration job to complete successful and having AWX instance up&running
Actual results
Actual result is having the awx-task pod stuck in
Init:0/3state with the logs ofinit-databasecontainer looping:Additional information
Looking at pod status "apparently" the job for migrating the DB did run successful:
But looking at
awx-migrationjob logs it seems the migration got somehow finished earlier so it did not complete:Logs of the migration job and describe
This is also confirmed by running
/bin/bash -c "! awx-manage showmigrations | grep '\[ \]'"insideawx-taskpod (init-databasecontainer)Workaround 1) Copy existing job
2) Remove the whole
statusblock at the bottom of yaml file and removeresourceVersion,uid(and any other uids inside the file)3) Delete currently present job
4) Apply previously saved and modified
awx-job.yamlA new job is getting created and runs, which should complete all the migrations this time
Operator Logs
No response