Atomic on provision_instance masks errors

[itdependsnetworks]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
• [X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Had an issue with permissions (due to some migration fun), but had to remove the @atomic decorator in order to see it.

Here is what it looked like:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 175, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/provision_instance.py", line 62, in handle
    self._register_hostname(options.get('hostname'), options.get('node_type'), options.get('uuid'))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/provision_instance.py", line 38, in _register_hostname
    (changed, instance) = Instance.objects.register(ip_address=os.environ.get('MY_POD_IP'), node_type='control', node_uuid=settings.SYSTEM_UUID)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/managers.py", line 188, in register
    instance = self.create(hostname=hostname, ip_address=ip_address, node_type=node_type, **create_defaults, **uuid_option)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/pglock.py", line 14, in advisory_lock
    yield internal_lock
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django_pglocks/__init__.py", line 80, in advisory_lock
    cursor.execute(command)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 83, in _execute
    self.db.validate_no_broken_transaction()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 531, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

Once we took off the atomic decorator, we saw:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
psycopg.errors.InsufficientPrivilege: permission denied for sequence main_instancelink_id_seq

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 175, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/provision_instance.py", line 61, in handle
    self._register_hostname(options.get('hostname'), options.get('node_type'), options.get('uuid'))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/provision_instance.py", line 38, in _register_hostname
    (changed, instance) = Instance.objects.register(ip_address=os.environ.get('MY_POD_IP'), node_type='control', node_uuid=settings.SYSTEM_UUID)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/managers.py", line 188, in register
    instance = self.create(hostname=hostname, ip_address=ip_address, node_type=node_type, **create_defaults, **uuid_option)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/models/ha.py", line 57, in save
    super(BaseModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 892, in save_base
    post_save.send(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/models/ha.py", line 570, in on_instance_saved
    instance.peers.add(*peers_addresses)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 1137, in add
    self._add_items(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 1432, in _add_items
    self.through._default_manager.using(db).bulk_create(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 803, in bulk_create
    returned_columns = self._batched_insert(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 1831, in _batched_insert
    self._insert(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: permission denied for sequence main_instancelink_id_seq

AWX version

23.8.1

Select the relevant components

• [ ] UI
• [ ] UI (tech preview)
• [ ] API
• [ ] Docs
• [ ] Collection
• [X] CLI
• [ ] Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Have a bad permission on main_instancelink_id_seq and run awx-manage provision_instance

Expected results

The error to publish with the underlying issue

Actual results

The error shows a yield internal_lock and You can't execute queries until the end of the 'atomic' block.

Additional information

No response

[fosterseth]

Can you explain a bit more of what you are trying to achieve? provision_instance is generally called programmatically by the operator

[itdependsnetworks]

Just looking to not mask the error, if the error is permission denied for sequence main_instancelink_id_seq it would be ideal to show that vs An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

It's not really clear to me why the atomic transaction is eating up the error, but is clear it did.

[itdependsnetworks]

just for clarity, here is the atomic decorator that was removed to show the underlying error https://github.com/ansible/awx/blob/9bb97dd6586224a44c5a8f3ae466e431c4fcef6c/awx/main/management/commands/provision_instance.py#L59

[itdependsnetworks]

Hmm, seems to be same issue described here https://code.djangoproject.com/ticket/23353, but either not fully complete or a regression.