search

ansible / awx-operator

An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖
https://www.github.com/ansible/awx
Apache License 2.0
1.26k stars 631 forks source link

Error when upgrading awx-operator from 2.1.0 to 2.2.0 #1673

Open racheldaguenethahn opened 10 months ago

racheldaguenethahn commented 10 months ago

Please confirm the following

Bug Summary

Hi,

I'm unable to upgrade AWX instance due to an issue on the migration of the managed postgresql DB. Something is wrong into the main_projectupdateevent table. Does someone know how to solve this? Table entries to delete? Logs from the awx-manager container of awx-operator 2.2.0 pod:

 TASK [Migrate the database if the K8s resources were updated.] ********************************
fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last):
File \"/usr/bin/awx-manage\", line 8, in <module>
  sys.exit(manage())
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage
  execute_from_command_line(sys.argv)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line
  utility.execute()
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv
  self.execute(*args, **cmd_options)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute
  output = self.handle(*args, **options)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper
  res = handle_func(*args, **kwargs)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle
  post_migrate_state = executor.migrate(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate
  state = self._migrate_all_forwards(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards
  state = self.apply_migration(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration
  state = migration.apply(state, schema_editor)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply
  operation.database_forwards(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards
  raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/bin/awx-manage\", line 8, in <module>", "    sys.exit(manage())", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage", "    execute_from_command_line(sys.argv)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", "    utility.execute()", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards", "    raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform:
Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations:
Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", "  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", "  Applying main.0184_django_indexes..."]}

Thank in advance Rachel

AWX Operator version

2.1.0

AWX version

22.2.0

Kubernetes platform

openshift

Kubernetes/Platform version

4.12.37

Modifications

no

Steps to reproduce

helm pull awx-operator/awx-operator --version 2.2.0 tar -zxf awx-operator-2.2.0.tgz

kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxbackups.awx.ansible.com.yaml kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxrestores.awx.ansible.com.yaml kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxs.awx.ansible.com.yaml

helm upgrade --install awx-operator ./awx-operator

Expected results

awx-operator is upgraded successfully AWX instance is upgraded successfully

Actual results

awx-operator is upgraded AWX instance is waiting for database migration

The following errors appears into awx-operator/awx-manager container

` TASK [Migrate the database if the K8s resources were updated.] **** fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last): File \"/usr/bin/awx-manage\", line 8, in sys.exit(manage()) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 200, in manage execute_from_command_line(sys.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py\", line 442, in execute_from_command_line utility.execute() File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py\", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv self.execute(*args, cmd_options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute output = self.handle(*args, *options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper res = handle_func(args, kwargs) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle post_migrate_state = executor.migrate( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate state = self._migrate_all_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards state = self.apply_migration( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration state = migration.apply(state, schema_editor) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply operation.database_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/awx-manage\", line 8, in ", " sys.exit(manage())", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 200, in manage", " execute_from_command_line(sys.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py\", line 442, in execute_from_command_line", " utility.execute()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py\", line 436, in execute", " self.fetch_command(subcommand).run_from_argv(self.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv", " self.execute(*args, cmd_options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", " output = self.handle(*args, *options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", " res = handle_func(args, kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", " post_migrate_state = executor.migrate(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", " state = self._migrate_all_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", " operation.database_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards", " raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform: Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations: Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", " Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", " Applying main.0184_django_indexes..."]}

`

Additional information

No response

Operator Logs

No response

jessicamack commented 10 months ago

Hello @racheldaguenethahn, we've had several fixes around similar issues. Are you able to upgrade to the latest version?

racheldaguenethahn commented 10 months ago

Hi @jessicamack , thank you for your answer. Yes I can upgrade to the latest version. Do you have a recommended upgrade path? Or can I upgrade directly from awx-operator 2.1.0 to awx-operator 2.10.0?

fosterseth commented 10 months ago

you should be able to upgrade from 2.1.0 to 2.10.0

racheldaguenethahn commented 10 months ago

Hi, I upgraded awx-operator from 2.1.0 to 2.10.0. The upgrade succeeded but now awx is blocked on a page saying that it's upgrading. When I try to access the URL, I'm redirected to https://privateurl/migrations_notran/ (see attachment) awx

From awx-web pod I can see the following error

192.168.84.1 - - [15/Jan/2024:08:11:27 +0000] "GET /static/media/logo-black.svg HTTP/1.1" 404 548 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" 2024/01/15 08:11:27 [error] 19#19: *92 open() "/var/lib/awx/public/static//media/logo-black.svg" failed (2: No such file or directory), client: 192.168.84.1, server: _, request: "GET /static/media/logo-black.svg HTTP/1.1", host: "privateurl", referrer: "https://privateurl/migrations_notran/" 192.168.84.1 - - [15/Jan/2024:08:11:37 +0000] "GET / HTTP/1.1" 302 5 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" [pid: 24|app: 0|req: 36/98] 192.168.84.1 () {64 vars in 1342 bytes} [Mon Jan 15 08:11:37 2024] GET / => generated 0 bytes in 77 msecs (HTTP/1.1 302) 5 headers in 220 bytes (1 switches on core 0) 192.168.84.1 - - [15/Jan/2024:08:11:38 +0000] "GET /migrations_notran/ HTTP/1.1" 200 2112 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" [pid: 22|app: 0|req: 36/99] 192.168.84.1 () {66 vars in 1378 bytes} [Mon Jan 15 08:11:37 2024] GET /migrations_notran/ => generated 2112 bytes in 71 msecs (HTTP/1.1 200) 7 headers in 261 bytes (1 switches on core 0)

jessicamack commented 10 months ago

The migrations_notrans URL is served when migrations haven't finished. This is usually because migrations are in progress but have not finished, however, if migrations error then it'll also serve that page. Can you provide the logs from the awx-task and postgres containers as well?

khorn7sk commented 10 months ago

@racheldaguenethahn I have the same issue, in my case pod awx-task- doesn't run(by security restrictions). Can you check? awx-task up and running?

racheldaguenethahn commented 10 months ago

@jessicamack I cannot provide you the logs from my production environment as I reverted to version 2.1.0.

However, I deployed another awx-operator 2.1.0 + awx instance into a sandbox cluster. This instance uses an unmanaged postgresql DB containing a backup of the production environment. Then, I upgraded this sandbox instance to 2.10.0 and I have the same result as in production => stucked on migration.

Find in attachment the logs of the postgres13 pod. postgres13.log

The logs from awx-task pod are not very useful...

[wait-for-migrations] Waiting for database migrations... [wait-for-migrations] Attempt 1 [wait-for-migrations] Waiting 0.5 seconds before next attempt [wait-for-migrations] Attempt 2 [wait-for-migrations] Waiting 1 seconds before next attempt [wait-for-migrations] Attempt 3 [wait-for-migrations] Waiting 2 seconds before next attempt [wait-for-migrations] Attempt 4 [wait-for-migrations] Waiting 4 seconds before next attempt [wait-for-migrations] Attempt 5 [wait-for-migrations] Waiting 8 seconds before next attempt [wait-for-migrations] Attempt 6 [wait-for-migrations] Waiting 16 seconds before next attempt [wait-for-migrations] Attempt 7 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 8 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 9 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 10 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 11 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 12 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 13 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 14

racheldaguenethahn commented 10 months ago

@khorn7sk yes the awx-task pod is up and running but waiting for the DB migration

racheldaguenethahn commented 9 months ago

Hi, I saw that there was a new version for awx-operator. I tried to upgrade awx-operator from 2.1.0 to 2.11.0

It still fails on the DB with the following issues

` 2024-02-06 10:47:47.449 UTC [76] ERROR: relation "conf_setting" does not exist at character 158 2024-02-06 10:47:47.449 UTC [76] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_HOST' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1 2024-02-06 10:47:47.451 UTC [76] ERROR: relation "conf_setting" does not exist at character 158 2024-02-06 10:47:47.451 UTC [76] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_PORT' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1 2024-02-06 10:47:47.940 UTC [78] ERROR: relation "conf_setting" does not exist at character 158 2024-02-06 10:47:47.940 UTC [78] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_HOST' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1 2024-02-06 10:47:47.941 UTC [78] ERROR: relation "conf_setting" does not exist at character 158 2024-02-06 10:47:47.941 UTC [78] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_PORT' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1 2024-02-06 10:47:55.306 UTC [81] ERROR: relation "main_unifiedjob" does not exist at character 1362 2024-02-06 10:47:55.306 UTC [81] STATEMENT: SELECT "main_unifiedjob"."id", "main_unifiedjob"."polymorphic_ctype_id", "main_unifiedjob"."modified", "main_unifiedjob"."description", "main_unifiedjob"."created_by_id", "main_unifiedjob"."modified_by_id", "main_unifiedjob"."name", "main_unifiedjob"."execution_environment_id", "main_unifiedjob"."old_pk", "main_unifiedjob"."emitted_events", "main_unifiedjob"."unified_job_template_id", "main_unifiedjob"."created", "main_unifiedjob"."launch_type", "main_unifiedjob"."schedule_id", "main_unifiedjob"."execution_node", "main_unifiedjob"."controller_node", "main_unifiedjob"."cancel_flag", "main_unifiedjob"."status", "main_unifiedjob"."failed", "main_unifiedjob"."started", "main_unifiedjob"."dependencies_processed", "main_unifiedjob"."finished", "main_unifiedjob"."canceled_on", "main_unifiedjob"."elapsed", "main_unifiedjob"."job_args", "main_unifiedjob"."job_cwd", "main_unifiedjob"."job_env", "main_unifiedjob"."job_explanation", "main_unifiedjob"."start_args", "main_unifiedjob"."result_traceback", "main_unifiedjob"."celery_task_id", "main_unifiedjob"."instance_group_id", "main_unifiedjob"."preferred_instance_groups_cache", "main_unifiedjob"."task_impact", "main_unifiedjob"."organization_id", "main_unifiedjob"."installed_collections", "main_unifiedjob"."ansible_version", "main_unifiedjob"."host_status_counts", "main_unifiedjob"."work_unit_id" FROM "main_unifiedjob" WHERE ("main_unifiedjob"."dependencies_processed" AND "main_unifiedjob"."status" IN ('pending', 'waiting', 'running') AND NOT ("main_unifiedjob"."launch_type" = 'sync') AND NOT ("main_unifiedjob"."polymorphic_ctype_id" = 68 AND "main_unifiedjob"."polymorphic_ctype_id" IS NOT NULL)) ORDER BY "main_unifiedjob"."created" ASC 2024-02-06 10:47:55.316 UTC [82] ERROR: relation "django_content_type" does not exist at character 106 2024-02-06 10:47:55.316 UTC [82] STATEMENT: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE ("django_content_type"."app_label" = 'main' AND "django_content_type"."model" = 'workflowapproval') LIMIT 21 2024-02-06 10:48:05.314 UTC [86] ERROR: relation "main_towerschedulestate" does not exist at character 91 2024-02-06 10:48:05.314 UTC [86] STATEMENT: SELECT "main_towerschedulestate"."id", "main_towerschedulestate"."schedule_last_run" FROM "main_towerschedulestate" WHERE "main_towerschedulestate"."id" = 1 LIMIT 21 2024-02-06 10:48:15.329 UTC [88] ERROR: relation "django_content_type" does not exist at character 106 2024-02-06 10:48:15.329 UTC [88] STATEMENT: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE ("django_content_type"."app_label" = 'main' AND "django_content_type"."model" = 'workflowapproval') LIMIT 21 2024-02-06 10:48:15.339 UTC [89] ERROR: relation "main_unifiedjob" does not exist at character 1362 2024-02-06 10:48:15.339 UTC [89] STATEMENT: SELECT "main_unifiedjob"."id", "main_unifiedjob"."polymorphic_ctype_id", "main_unifiedjob"."modified", "main_unifiedjob"."description", "main_unifiedjob"."created_by_id", "main_unifiedjob"."modified_by_id", "main_unifiedjob"."name", "main_unifiedjob"."execution_environment_id", "main_unifiedjob"."old_pk", "main_unifiedjob"."emitted_events", "main_unifiedjob"."unified_job_template_id", "main_unifiedjob"."created", "main_unifiedjob"."launch_type", "main_unifiedjob"."schedule_id", "main_unifiedjob"."execution_node", "main_unifiedjob"."controller_node", "main_unifiedjob"."cancel_flag", "main_unifiedjob"."status", "main_unifiedjob"."failed", "main_unifiedjob"."started", "main_unifiedjob"."dependencies_processed", "main_unifiedjob"."finished", "main_unifiedjob"."canceled_on", "main_unifiedjob"."elapsed", "main_unifiedjob"."job_args", "main_unifiedjob"."job_cwd", "main_unifiedjob"."job_env", "main_unifiedjob"."job_explanation", "main_unifiedjob"."start_args", "main_unifiedjob"."result_traceback", "main_unifiedjob"."celery_task_id", "main_unifiedjob"."instance_group_id", "main_unifiedjob"."preferred_instance_groups_cache", "main_unifiedjob"."task_impact", "main_unifiedjob"."organization_id", "main_unifiedjob"."installed_collections", "main_unifiedjob"."ansible_version", "main_unifiedjob"."host_status_counts", "main_unifiedjob"."work_unit_id" FROM "main_unifiedjob" WHERE (NOT "main_unifiedjob"."dependencies_processed" AND "main_unifiedjob"."status" IN ('pending') AND NOT ("main_unifiedjob"."launch_type" = 'sync') AND NOT ("main_unifiedjob"."polymorphic_ctype_id" = 68 AND "main_unifiedjob"."polymorphic_ctype_id" IS NOT NULL)) ORDER BY "main_unifiedjob"."created" ASC 2024-02-06 10:48:35.353 UTC [91] ERROR: relation "main_instance" does not exist at character 666 2024-02-06 10:48:35.353 UTC [91] STATEMENT: SELECT "main_instance"."id", "main_instance"."uuid", "main_instance"."hostname", "main_instance"."ip_address", "main_instance"."created", "main_instance"."modified", "main_instance"."version", "main_instance"."cpu", "main_instance"."memory", "main_instance"."errors", "main_instance"."last_seen", "main_instance"."health_check_started", "main_instance"."last_health_check", "main_instance"."capacity", "main_instance"."capacity_adjustment", "main_instance"."enabled", "main_instance"."managed_by_policy", "main_instance"."cpu_capacity", "main_instance"."mem_capacity", "main_instance"."node_type", "main_instance"."node_state", "main_instance"."listener_port" FROM "main_instance" WHERE "main_instance"."node_state" IN ('ready', 'unavailable', 'installed') ORDER BY "main_instance"."hostname" ASC 2024-02-06 10:48:35.373 UTC [94] ERROR: relation "main_instancegroup" does not exist at character 666 2024-02-06 10:48:35.373 UTC [94] STATEMENT: DECLARE "_django_curs_139787606607680_sync_1" NO SCROLL CURSOR WITH HOLD FOR SELECT "main_instancegroup"."id", "main_instancegroup"."name", "main_instancegroup"."created", "main_instancegroup"."modified", "main_instancegroup"."is_container_group", "main_instancegroup"."credential_id", "main_instancegroup"."pod_spec_override", "main_instancegroup"."admin_role_id", "main_instancegroup"."use_role_id", "main_instancegroup"."read_role_id", "main_instancegroup"."max_concurrent_jobs", "main_instancegroup"."max_forks", "main_instancegroup"."policy_instance_percentage", "main_instancegroup"."policy_instance_minimum", "main_instancegroup"."policy_instance_list" FROM "main_instancegroup" WHERE "main_instancegroup"."is_container_group" 2024-02-06 10:48:35.373 UTC [92] ERROR: relation "main_towerschedulestate" does not exist at character 91 2024-02-06 10:48:35.373 UTC [92] STATEMENT: SELECT "main_towerschedulestate"."id", "main_towerschedulestate"."schedule_last_run" FROM "main_towerschedulestate" WHERE "main_towerschedulestate"."id" = 1 LIMIT 21 2024-02-06 10:48:35.375 UTC [94] ERROR: cursor "_django_curs_139787606607680_sync_1" does not exist 2024-02-06 10:48:35.375 UTC [94] STATEMENT: CLOSE "_django_curs_139787606607680_sync_1"

`

racheldaguenethahn commented 7 months ago

Hello, I still have this issue. Current awx-operator version is 2.1.0. When trying to upgrade my AWX instance by upgrading the operator to 2.2.0, it fails.

Going into the awx-web pod to run awx-manage command line I have the same issue:

bash-5.1$ awx-manage showmigrations auth [X] 0001_initial [X] 0002_alter_permission_name_max_length [X] 0003_alter_user_email_max_length [X] 0004_alter_user_username_opts [X] 0005_alter_user_last_login_null [X] 0006_require_contenttypes_0002 [X] 0007_alter_validators_add_error_messages [X] 0008_alter_user_username_max_length [X] 0009_alter_user_last_name_max_length [X] 0010_alter_group_name_max_length [X] 0011_update_proxy_permissions [X] 0012_alter_user_first_name_max_length conf [X] 0001_initial [X] 0002_v310_copy_tower_settings [X] 0003_v310_JSONField_changes [X] 0004_v320_reencrypt [X] 0005_v330_rename_two_session_settings [X] 0006_v331_ldap_group_type [X] 0007_v380_rename_more_settings [X] 0008_subscriptions [X] 0009_rename_proot_settings contenttypes [X] 0001_initial [X] 0002_remove_content_type_name main [X] 0001_initial [X] 0002_squashed_v300_release (18 squashed migrations) [X] 0003_squashed_v300_v303_updates (9 squashed migrations) [X] 0004_squashed_v310_release (6 squashed migrations) [X] 0005_squashed_v310_v313_updates (3 squashed migrations) [X] 0006_v320_release [X] 0007_v320_data_migrations [X] 0008_v320_drop_v1_credential_fields [X] 0009_v322_add_setting_field_for_activity_stream [X] 0010_v322_add_ovirt4_tower_inventory [X] 0011_v322_encrypt_survey_passwords [X] 0012_v322_update_cred_types [X] 0013_v330_multi_credential [X] 0014_v330_saved_launchtime_configs [X] 0015_v330_blank_start_args [X] 0016_v330_non_blank_workflow [X] 0017_v330_move_deprecated_stdout [X] 0018_v330_add_additional_stdout_events [X] 0019_v330_custom_virtualenv [X] 0020_v330_instancegroup_policies [X] 0021_v330_declare_new_rbac_roles [X] 0022_v330_create_new_rbac_roles [X] 0023_v330_inventory_multicred [X] 0024_v330_create_user_session_membership [X] 0025_v330_add_oauth_activity_stream_registrar [X] 0026_v330_delete_authtoken [X] 0027_v330_emitted_events [X] 0028_v330_add_tower_verify [X] 0030_v330_modify_application [X] 0031_v330_encrypt_oauth2_secret [X] 0032_v330_polymorphic_delete [X] 0033_v330_oauth_help_text [X] 0034_v330_delete_user_role [X] 0035_v330_more_oauth2_help_text [X] 0036_v330_credtype_remove_become_methods [X] 0037_v330_remove_legacy_fact_cleanup [X] 0038_v330_add_deleted_activitystream_actor [X] 0039_v330_custom_venv_help_text [X] 0040_v330_unifiedjob_controller_node [X] 0041_v330_update_oauth_refreshtoken [X] 0042_v330_org_member_role_deparent [X] 0043_v330_oauth2accesstoken_modified [X] 0044_v330_add_inventory_update_inventory [X] 0045_v330_instance_managed_by_policy [X] 0046_v330_remove_client_credentials_grant [X] 0047_v330_activitystream_instance [X] 0048_v330_django_created_modified_by_model_name [X] 0049_v330_validate_instance_capacity_adjustment [X] 0050_v340_drop_celery_tables [X] 0051_v340_job_slicing [X] 0052_v340_remove_project_scm_delete_on_next_update [X] 0053_v340_workflow_inventory [X] 0054_v340_workflow_convergence [X] 0055_v340_add_grafana_notification [X] 0056_v350_custom_venv_history [X] 0057_v350_remove_become_method_type [X] 0058_v350_remove_limit_limit [X] 0059_v350_remove_adhoc_limit [X] 0060_v350_update_schedule_uniqueness_constraint [X] 0061_v350_track_native_credentialtype_source [X] 0062_v350_new_playbook_stats [X] 0063_v350_org_host_limits [X] 0064_v350_analytics_state [X] 0065_v350_index_job_status [X] 0066_v350_inventorysource_custom_virtualenv [X] 0067_v350_credential_plugins [X] 0068_v350_index_event_created [X] 0069_v350_generate_unique_install_uuid [X] 0070_v350_gce_instance_id [X] 0071_v350_remove_system_tracking [X] 0072_v350_deprecate_fields [X] 0073_v360_create_instance_group_m2m [X] 0074_v360_migrate_instance_group_relations [X] 0075_v360_remove_old_instance_group_relations [X] 0076_v360_add_new_instance_group_relations [X] 0077_v360_add_default_orderings [X] 0078_v360_clear_sessions_tokens_jt [X] 0079_v360_rm_implicit_oauth2_apps [X] 0080_v360_replace_job_origin [X] 0081_v360_notify_on_start [X] 0082_v360_webhook_http_method [X] 0083_v360_job_branch_override [X] 0084_v360_token_description [X] 0085_v360_add_notificationtemplate_messages [X] 0086_v360_workflow_approval [X] 0087_v360_update_credential_injector_help_text [X] 0088_v360_dashboard_optimizations [X] 0089_v360_new_job_event_types [X] 0090_v360_WFJT_prompts [X] 0091_v360_approval_node_notifications [X] 0092_v360_webhook_mixin [X] 0093_v360_personal_access_tokens [X] 0094_v360_webhook_mixin2 [X] 0095_v360_increase_instance_version_length [X] 0096_v360_container_groups [X] 0097_v360_workflowapproval_approved_or_denied_by [X] 0098_v360_rename_cyberark_aim_credential_type [X] 0099_v361_license_cleanup [X] 0100_v370_projectupdate_job_tags [X] 0101_v370_generate_new_uuids_for_iso_nodes [X] 0102_v370_unifiedjob_canceled [X] 0103_v370_remove_computed_fields [X] 0104_v370_cleanup_old_scan_jts [X] 0105_v370_remove_jobevent_parent_and_hosts [X] 0106_v370_remove_inventory_groups_with_active_failures [X] 0107_v370_workflow_convergence_api_toggle [X] 0108_v370_unifiedjob_dependencies_processed [X] 0109_v370_job_template_organization_field [X] 0110_v370_instance_ip_address [X] 0111_v370_delete_channelgroup [X] 0112_v370_workflow_node_identifier [X] 0113_v370_event_bigint [X] 0114_v370_remove_deprecated_manual_inventory_sources [X] 0115_v370_schedule_set_null [X] 0116_v400_remove_hipchat_notifications [X] 0117_v400_remove_cloudforms_inventory [X] 0118_add_remote_archive_scm_type [X] 0119_inventory_plugins [X] 0120_galaxy_credentials [X] 0121_delete_toweranalyticsstate [X] 0122_really_remove_cloudforms_inventory [X] 0123_drop_hg_support [X] 0124_execution_environments [X] 0125_more_ee_modeling_changes [X] 0126_executionenvironment_container_options [X] 0127_reset_pod_spec_override [X] 0128_organiaztion_read_roles_ee_admin [X] 0129_unifiedjob_installed_collections [X] 0130_ee_polymorphic_set_null [X] 0131_undo_org_polymorphic_ee [X] 0132_instancegroup_is_container_group [X] 0133_centrify_vault_credtype [X] 0134_unifiedjob_ansible_version [X] 0135_schedule_sort_fallback_to_id [X] 0136_scm_track_submodules [X] 0137_custom_inventory_scripts_removal_data [X] 0138_custom_inventory_scripts_removal [X] 0139_isolated_removal [X] 0140_rename [X] 0141_remove_isolated_instances [X] 0142_update_ee_image_field_description [X] 0143_hostmetric [X] 0144_event_partitions [X] 0145_deregister_managed_ee_objs [X] 0146_add_insights_inventory [X] 0147_validate_ee_image_field [X] 0148_unifiedjob_receptor_unit_id [X] 0149_remove_inventory_insights_credential [X] 0150_rename_inv_sources_inv_updates [X] 0151_rename_managed_by_tower [X] 0152_instance_node_type [X] 0153_instance_last_seen [X] 0154_set_default_uuid [X] 0155_improved_health_check [X] 0156_capture_mesh_topology [X] 0157_inventory_labels [X] 0158_make_instance_cpu_decimal [X] 0159_deprecate_inventory_source_UoPU_field [X] 0160_alter_schedule_rrule [X] 0161_unifiedjob_host_status_counts [X] 0162_alter_unifiedjob_dependent_jobs [X] 0163_convert_job_tags_to_textfield [X] 0164_remove_inventorysource_update_on_project_update [X] 0165_task_manager_refactor [X] 0166_alter_jobevent_host [X] 0167_project_signature_validation_credential [X] 0168_inventoryupdate_scm_revision [X] 0169_jt_prompt_everything_on_launch [X] 0170_node_and_link_state [X] 0171_add_health_check_started [X] 0172_prevent_instance_fallback [X] 0173_instancegroup_max_limits [X] 0174_ensure_org_ee_admin_roles [X] 0175_workflowjob_is_bulk_job [X] 0176_inventorysource_scm_branch [X] 0177_instance_group_role_addition [X] 0178_instance_group_admin_migration [X] 0179_change_cyberark_plugin_names [X] 0180_add_hostmetric_fields [X] 0181_hostmetricsummarymonthly [X] 0182_constructed_inventory [X] 0183_pre_django_upgrade [ ] 0184_django_indexes oauth2_provider [X] 0001_initial [X] 0002_auto_20190406_1805 [X] 0003_auto_20201211_1314 [X] 0004_auto_20200902_2022 [X] 0005_auto_20211222_2352 sessions [X] 0001_initial sites [X] 0001_initial [X] 0002_alter_domain_unique social_django [X] 0001_initial (2 squashed migrations) [X] 0002_add_related_name (2 squashed migrations) [X] 0003_alter_email_max_length (2 squashed migrations) [X] 0004_auto_20160423_0400 (2 squashed migrations) [X] 0005_auto_20160727_2333 (1 squashed migrations) [X] 0006_partial [X] 0007_code_timestamp [X] 0008_partial_timestamp [X] 0009_auto_20191118_0520 [X] 0010_uid_db_index sso [X] 0001_initial [X] 0002_expand_provider_options [X] 0003_convert_saml_string_to_list taggit [X] 0001_initial [X] 0002_auto_20150616_2121 [X] 0003_taggeditem_add_unique_index [X] 0004_alter_taggeditem_content_type_alter_taggeditem_tag [X] 0005_auto_20220424_2025

Only 0184_django_indexes is failing

bash-5.1$ awx-manage migrate main Operations to perform: Apply all migrations: main Running migrations: Applying main.0184_django_indexes...Traceback (most recent call last): File "/usr/bin/awx-manage", line 8, in sys.exit(manage()) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py", line 200, in manage execute_from_command_line(sys.argv) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 442, in execute_from_command_line utility.execute() File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv self.execute(*args, cmd_options) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute output = self.handle(*args, *options) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper res = handle_func(args, kwargs) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle post_migrate_state = executor.migrate( File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards state = self.apply_migration( File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration state = migration.apply(state, schema_editor) File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply operation.database_forwards( File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py", line 977, in database_forwards raise ValueError( ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).

Any ideas?