search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.9k stars 3.4k forks source link

no partition of relation "main_projectupdateevent" found #10744

Open phemmer opened 3 years ago

phemmer commented 3 years ago

Please confirm the following

Summary

After changing the execution environment of a inventory source, I received the following exception when trying to perform an inventory sync.

django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

AWX version

19.2.2

Installation method

minikube

Modifications

no

Ansible version

2.10.11

Operating system

Debian Buster

Web browser

Chrome

Steps to reproduce

Not sure. The last thing I did prior to this was to change the execution environment on an inventory source.

Expected results

Jobs to run

Actual results

2021-07-23 17:45:42,050 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.dispatch task aab78528-b800-495a-8ce8-9355a64689ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-07-23 17:45:42,053 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.scheduler Running task manager.
2021-07-23 17:45:42,065 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.scheduler Starting Scheduler
2021-07-23 17:45:42,118 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle inventoryupdate-383 acknowledged
2021-07-23 17:45:42,169 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.models.inventory Going to update inventory computed fields, pk=2
2021-07-23 17:45:42,179 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.models.inventory Finished updating inventory computed fields, pk=2, in 0.010 seconds
2021-07-23 17:45:42,180 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle inventoryupdate-383 waiting
2021-07-23 17:45:42,180 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.scheduler inventory_update 383 (waiting) consumed 1 capacity units from default with prior total of 0
2021-07-23 17:45:42,181 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.scheduler Finishing Scheduler
2021-07-23 17:45:42,200 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.dispatch task f4767f62-4d57-4b8b-b3a9-d1fdeb13d2ad starting awx.main.tasks.RunInventoryUpdate(*[383])
2021-07-23 17:45:42,280 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.models.inventory Going to update inventory computed fields, pk=2
2021-07-23 17:45:42,292 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.main.models.inventory Finished updating inventory computed fields, pk=2, in 0.011 seconds
2021-07-23 17:45:42,316 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle inventoryupdate-383 pre run
2021-07-23 17:45:42,339 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle projectupdate-384 created
2021-07-23 17:45:42,411 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle projectupdate-384 pre run
2021-07-23 17:45:42,421 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle projectupdate-384 preparing playbook
2021-07-23 17:45:42,452 DEBUG    [3f9134a3faf445ae8a3694e5d316ff83] awx.analytics.job_lifecycle projectupdate-384 running playbook

2021-07-23 17:45:43,487 DEBUG    [-] awx.main.commands.run_callback_receiver ProjectUpdateEvent.objects.bulk_create(1)
2021-07-23 17:45:43,487 DEBUG    [-] awx.main.commands.run_callback_receiver ProjectUpdateEvent.objects.bulk_create(1)
2021-07-23 17:45:43,502 ERROR    [-] awx.main.commands.run_callback_receiver Database Error Saving Job Event
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/dispatch/worker/callback.py", line 130, in flush
    cls.objects.bulk_create(events)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 474, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1202, in _batched_insert
    inserted_id = self._insert(
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/dispatch/worker/callback.py", line 139, in flush
    e.save()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/models/base.py", line 176, in save
    super(CreatedModifiedModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

2021-07-23 17:45:43,502 ERROR    [-] awx.main.commands.run_callback_receiver Database Error Saving Job Event
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/dispatch/worker/callback.py", line 130, in flush
    cls.objects.bulk_create(events)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 474, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1202, in _batched_insert
    inserted_id = self._insert(
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/dispatch/worker/callback.py", line 139, in flush
    e.save()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/models/base.py", line 176, in save
    super(CreatedModifiedModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row
DETAIL:  Partition key of the failing row contains (job_created) = (2021-07-23 17:45:42.319417+00).

Additional information

No response

shanemcd commented 3 years ago

This might be fixed in the next release of AWX. I'll try to follow up here when it goes live so you can test it out, or you can build your own images from devel.

kenichimilo commented 3 years ago

@shanemcd i suppose the fix hasn't been committed yet.. as building the current version does not provide a fix.. Or if there is a workaround for this it would be more than welcome.. thnxs

HOSTED-POWER commented 3 years ago

Also seeing this error often in the logs:

psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row

kenichimilo commented 3 years ago

The fix seems to have introduced another bug described in https://github.com/ansible/awx/issues/10838, the public schema has been hardcoded..,

kenichimilo commented 2 years ago

hi guys, Is there a specific way of manually calling this migration script.. or have it run till the end No errors in operator when upgrading or deploying.. https://github.com/ansible/awx/blob/0f6e221c14ec60fd6e492e583676511e040f469b/awx/main/migrations/0144_event_partitions.py Each time i try a new upgrade i get the same File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: no partition of relation "main_projectupdateevent" found for row DETAIL: Partition key of the failing row contains (job_created) = (2021-10-05 13:02:45.157503+00). During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.CheckViolation: no partition of relation "main_projectupdateevent" found for row DETAIL: Partition key of the failing row contains (job_created) = (2021-10-05 13:02:45.157503+00).

migrating from awx 7.1.0 , operator 0.2.0 externaldb migrating to: 19.4.0 ,operator 0.14.0 The migration occurs but only half way.. unpartitioned* tables are created, though even by manually purging the original tables and renaming the unpartitioned ones, you end up having tables without some columns like job_created.   | django.db.utils.ProgrammingError: column main_projectupdateevent.job_created does not exist   | LINE 1: ..._update_id" FROM "main_projectupdateevent" WHERE ("main_proj..

shanemcd commented 2 years ago

I think this might have been fixed by https://github.com/ansible/awx/pull/11241

kenichimilo commented 2 years ago

Thanks @shanemcd seems like the commit/merge was done after the build 19.4.0> About when could we expect a new build and if so could we make sure that 10838, is included.. Becomes a hustle if you are using an external db. As not good practice to use the public schema..

kenichimilo commented 2 years ago

https://github.com/ansible/awx/blob/7fdf491c059552eac65b33ca2d106540c3e5f8a6/awx/main/management/commands/cleanup_jobs.py#L108

This is what i was talking about.. unfortunately build went out without it being fixed

mamercad commented 1 year ago

I'm seeing this on 20.0.1.