search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.9k stars 3.4k forks source link

Bulk deletion of jobs leads to internal error #11165

Open nixocio opened 2 years ago

nixocio commented 2 years ago

Please confirm the following

Summary

Bulk deletion of jobs leads to internal error

AWX version

devel

Installation method

docker for mac

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

1 - Trigger a miscellaneous of several jobs, like, inventory source sync, workflow, and job templates. 2 - Select all of the jobs on the first page to be deleted. 3 - Delete them

Expected results

All jobs to be deleted.

Actual results

image

Additional information

It is seem like workflow job that trigger then job template to run was deleted first, but I was not able to conclude that was the reason of the internal server failure.


tools_awx_1     | awx-uwsgi stdout | 172.19.0.1 DELETE /api/v2/project_updates/130/ - HTTP/1.1 204
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | 2021-09-29 13:02:34,402 ERROR    [189b3bfd] django.request Internal Server Error: /api/v2/jobs/128/
tools_awx_1     | awx-uwsgi stdout | Traceback (most recent call last):
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout | psycopg2.errors.ForeignKeyViolation: insert or update on table "main_host" violates foreign key constraint "main_host_last_job_host_summar_b8bd727d_fk_main_jobh"
tools_awx_1     | awx-uwsgi stdout | DETAIL:  Key (last_job_host_summary_id)=(45) is not present in table "main_jobhostsummary".
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | The above exception was the direct cause of the following exception:
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | Traceback (most recent call last):
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
tools_awx_1     | awx-uwsgi stdout |     response = get_response(request)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
tools_awx_1     | awx-uwsgi stdout |     response = self.process_exception_by_middleware(e, request)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
tools_awx_1     | awx-uwsgi stdout |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
tools_awx_1     | awx-uwsgi stdout |   File "/usr/lib64/python3.8/contextlib.py", line 75, in inner
tools_awx_1     | awx-uwsgi stdout |     return func(*args, **kwds)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/transaction.py", line 240, in __exit__
tools_awx_1     | awx-uwsgi stdout |     connection.commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 262, in commit
tools_awx_1     | awx-uwsgi stdout |     self._commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
tools_awx_1     | awx-uwsgi stdout |     raise dj_exc_value.with_traceback(traceback) from exc_value
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout | django.db.utils.IntegrityError: insert or update on table "main_host" violates foreign key constraint "main_host_last_job_host_summar_b8bd727d_fk_main_jobh"
tools_awx_1     | awx-uwsgi stdout | DETAIL:  Key (last_job_host_summary_id)=(45) is not present in table "main_jobhostsummary".
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | 2021-09-29 13:02:34,402 ERROR    [189b3bfd] django.request Internal Server Error: /api/v2/jobs/128/
tools_awx_1     | awx-uwsgi stdout | Traceback (most recent call last):
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout | psycopg2.errors.ForeignKeyViolation: insert or update on table "main_host" violates foreign key constraint "main_host_last_job_host_summar_b8bd727d_fk_main_jobh"
tools_awx_1     | awx-uwsgi stdout | DETAIL:  Key (last_job_host_summary_id)=(45) is not present in table "main_jobhostsummary".
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | The above exception was the direct cause of the following exception:
tools_awx_1     | awx-uwsgi stdout |
tools_awx_1     | awx-uwsgi stdout | Traceback (most recent call last):
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
tools_awx_1     | awx-uwsgi stdout |     response = get_response(request)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
tools_awx_1     | awx-uwsgi stdout |     response = self.process_exception_by_middleware(e, request)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
tools_awx_1     | awx-uwsgi stdout |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
tools_awx_1     | awx-uwsgi stdout |   File "/usr/lib64/python3.8/contextlib.py", line 75, in inner
tools_awx_1     | awx-uwsgi stdout |     return func(*args, **kwds)
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/transaction.py", line 240, in __exit__
tools_awx_1     | awx-uwsgi stdout |     connection.commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 262, in commit
tools_awx_1     | awx-uwsgi stdout |     self._commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
tools_awx_1     | awx-uwsgi stdout |     raise dj_exc_value.with_traceback(traceback) from exc_value
tools_awx_1     | awx-uwsgi stdout |   File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 240, in _commit
tools_awx_1     | awx-uwsgi stdout |     return self.connection.commit()
tools_awx_1     | awx-uwsgi stdout | django.db.utils.IntegrityError: insert or update on table "main_host" violates foreign key constraint "main_host_last_job_host_summar_b8bd727d_fk_main_jobh"
tools_awx_1     | awx-uwsgi stdout | DETAIL:  Key (last_job_host_summary_id)=(45) is not present in table "main_jobhostsummary".
tools_awx_1     | awx-uwsgi stdout |
chrismeyersfsu commented 2 years ago

This is kind of a larger problem with AWX API. Our API request/response cycle happens in a transaction, so that is good. The bad news is that it really only protects the resource that we are directly acting upon. i.e. DELETE /api/v2/project_updates/130/ In our request handling, it's "safe" to do all kinds of update/delete to project update 130. It's not "safe" to do things to it's related objects. We should sprinkle in select_for_update() to grab a lock on the related resources that we intend to modify, before modifying them. The disadvantage is that obtaining these locks could take time. If they take too much time, we might consider backgrounding the operating via a .delay() call like we do with inventory delete.

This specific case may be more complicated than a simple sleect_for_update() but it's what comes to mind when I see this sort of problem.