AWX 9.2 - VMware dynamic inventory error on refresh

[linuxcube]

ISSUE TYPE

• Bug Report

SUMMARY

When running the vCenter dynamic inventory job to refresh the inventory objects within AWX and error occurs.

ENVIRONMENT

• AWX version: 9.2.0
• AWX install method: docker on linux
• Ansible version: 2.9.3
• Operating System: RHEL 7.7
• Web Browser: Chrome, Firefox, IE

STEPS TO REPRODUCE

When running Dynamic inventory job to update the vCenter inventory form our vCenter we are getting job failure.

EXPECTED RESULTS

We expect the job to finish without issues.

ACTUAL RESULTS

The following error were observed:

    1.971 INFO     Updating inventory 3: vCenter-DI
    2.386 DEBUG    Using base command: python /usr/bin/ansible-inventory -i /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/plugins/inventory/vmware_inventory.py --playbook-dir /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/plugins/inventory -vvvvv
    2.386 INFO     Reading Ansible inventory source: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/plugins/inventory/vmware_inventory.py
    2.388 INFO     Using VIRTUAL_ENV: /var/lib/awx/venv/ansible
    2.388 INFO     Using PATH: /var/lib/awx/venv/ansible/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    2.388 INFO     Using PYTHONPATH: /var/lib/awx/venv/ansible/lib/python3.6/site-packages:
 1580.088 ERROR    ansible-inventory 2.9.3
 1580.088 ERROR      config file = /etc/ansible/ansible.cfg
 1580.088 ERROR      configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
 1580.088 ERROR      ansible python module location = /usr/lib/python3.6/site-packages/ansible
 1580.088 ERROR      executable location = /usr/bin/ansible-inventory
 1580.088 ERROR      python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
 1580.088 ERROR    Using /etc/ansible/ansible.cfg as config file
 1580.088 ERROR    setting up inventory plugins
 1580.088 ERROR    Parsed /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/plugins/inventory/vmware_inventory.py inventory source with script plugin
 1580.088 ERROR    [WARNING]: No setting was provided for required configuration plugin_type:
 1580.088 ERROR    cache plugin: redis setting: _uri
 1580.088 ERROR    
 1580.788 DEBUG    Finished loading from source: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/plugins/inventory/vmware_inventory.py
 1580.788 INFO     Processing JSON output...
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "main_host_name_inventory_id_45aecd68_uniq"
DETAIL:  Key (name, inventory_id)=(fcebsdbdev01_clone, 3) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django_pglocks/__init__.py", line 74, in advisory_lock
    yield acquired
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/utils/pglock.py", line 14, in advisory_lock
    yield internal_lock
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 898, in load_into_database
    self._create_update_hosts()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 772, in _create_update_hosts
    self._update_db_host_from_mem_host(db_host, mem_host)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 708, in _update_db_host_from_mem_host
    db_host.save(update_fields=update_fields)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/inventory.py", line 597, in save
    super(Host, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/base.py", line 319, in save
    super(PrimordialModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/base.py", line 164, in save
    super(CreatedModifiedModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1462, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "main_host_name_inventory_id_45aecd68_uniq"
DETAIL:  Key (name, inventory_id)=(srv01_clone, 3) already exists.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/__init__.py", line 152, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1157, in handle
    raise exc
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1092, in handle
    self.load_into_database()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 900, in load_into_database
    self._create_update_group_hosts()
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/utils/pglock.py", line 14, in advisory_lock
    yield internal_lock
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django_pglocks/__init__.py", line 80, in advisory_lock
    cursor.execute(command)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in _execute
    self.db.validate_no_broken_transaction()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 438, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

ADDITIONAL INFORMATION

No additional details were provided by the job output.

[wenottingham]

key point of error:

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "main_host_name_inventory_id_45aecd68_uniq" DETAIL: Key (name, inventory_id)=(fcebsdbdev01_clone, 3) already exists.

[wenottingham]

Can you paste your configuration for all sources for this inventory?

[linuxcube]

Can you paste your configuration for all sources for this inventory?

I am not sure in your request, this is a basic source with "Vcenter server" and basic credentials define for the vcenter. Can you please be more detail on the ask? Thank you.

[wenottingham]

Is there only one source in the inventory? Are you using any filters? Do you have other inventories in the same organization talking to the same VCenter?

Essentially, we want to characterize when this may happen, because it works in our testing.

[linuxcube]

Is there only one source in the inventory? Are you using any filters? Do you have other inventories in the same organization talking to the same VCenter?

Essentially, we want to characterize when this may happen, because it works in our testing.

I think I get your question now, there is only 1 source of inventory. I am not using filters in any way/shape/form. I have other inventory define in the organization, but they are talking to RedHat satellite, and those do not experienced any issues with nightly syncing, only the vcenter one is experiencing this issue.

[wenottingham]

Which checkboxes (overwrite, overwrite vars, etc.) are set on the source?

[linuxcube]

zero check boxes checked on the source for vcenter dynamic inventory

[wenottingham]

If you check 'overwrite', does it work?

[AlanCoding]

From a quick look at the inventory import code, I think we may be missing the 2nd part of this deadlock because https://github.com/Xof/django-pglocks/issues/21 causes the error to get swallowed. If we get a reproducer, we should start out by rearranging these context managers to give accurate tracebacks.

[linuxcube]

I just tried with overwrite....same outcome.

On Fri, Mar 6, 2020, 1:29 PM Bill Nottingham notifications@github.com wrote:

If you check 'overwrite', does it work?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ansible/awx/issues/6136?email_source=notifications&email_token=AD7QQLEM7NW62DFKOLYJQQ3RGFTLZA5CNFSM4K74Q4OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOC5IBA#issuecomment-595973124, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7QQLHHD332LZCD32JBVQLRGFTLZANCNFSM4K74Q4OA .

[linuxcube]

From a quick look at the inventory import code, I think we may be missing the 2nd part of this deadlock because Xof/django-pglocks#21 causes the error to get swallowed. If we get a reproducer, we should start out by rearranging these context managers to give accurate tracebacks.

Hi AlanCoding, please let me know how I can help and I will share what I have in the debug logs.

[AlanCoding]

The most valuable information might be "what changed?" It would help to get some scale numbers on:

• number of hosts in the inventory
• number of groups
• how many hosts or groups might have been added or removed since the last import?

We saw mention of (name, inventory_id)=(fcebsdbdev01_clone, 3) and (name, inventory_id)=(srv01_clone, 3), does the _clone thing mean anything to you regarding how these were provisioned? Do hosts with the names fcebsdbdev01_clone or srv01_clone already exist in the inventory?

[linuxcube]

Those hosts do exists in the inventory, should I remove them and resync the source?

[AlanCoding]

These questions might get a little weirder, but they could be helpful in building a reproducer.

Do the host variables have an entry under config.instanceuuid? That's not the same as another host without the _clone in its name, is it?

Also, I'm interested if it is truly recognized as a part of that inventory source. If you run awx-manage shell_plus on the server, you might be able to deduce this by running something like Host.objects.get(name='fcebsdbdev01_clone', inventory_id=3).inventory_sources.all(), and then we want to know if it's an empty list or has the vmware inventory source.

It would also be worth a try to delete those hosts and see if it runs, both to get you unblocked and to understand the problem a little better.

[zappbrannigan34]

AWX 19.2.2 same issue

[zappbrannigan34]

`Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "main_host_name_inventory_id_45aecd68_uniq" DETAIL: Key (name, inventory_id)=(ixia4-vblade0301_564de870-a2c5-ac87-62d2-8585de8ff875, 5) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2632, in post_run_hook cmd.perform_update(options, data, inventory_update) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/management/commands/inventory_import.py", line 977, in perform_update self.load_into_database() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/management/commands/inventory_import.py", line 735, in load_into_database self._create_update_hosts() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/management/commands/inventory_import.py", line 613, in _create_update_hosts self._update_db_host_from_mem_host(db_host, mem_host) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/management/commands/inventory_import.py", line 549, in _update_db_host_from_mem_host db_host.save(update_fields=update_fields) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/models/inventory.py", line 569, in save super(Host, self).save(*args, kwargs) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/models/base.py", line 329, in save super(PrimordialModel, self).save(*args, *kwargs) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/models/base.py", line 176, in save super(CreatedModifiedModel, self).save(args, kwargs) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 743, in save self.save_base(using=using, force_insert=force_insert, File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 780, in save_base updated = self._save_table( File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 853, in _save_table updated = self._do_update(base_qs, using, pk_val, values, update_fields, File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/base.py", line 903, in _do_update return filtered._update(values) > 0 File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 760, in _update return query.get_compiler(self.db).execute_sql(CURSOR) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql cursor = super().execute_sql(result_type) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: duplicate key value violates unique constraint "main_host_name_inventory_id_45aecd68_uniq" DETAIL: Key (name, inventory_id)=(ixia4-vblade0301_564de870-a2c5-ac87-62d2-8585de8ff875, 5) already exists. `

[zappbrannigan34]

AWX 19.3.0 same

[ghost]

Same issue on AWX 12.0.0. In our case I think it's caused when we switch between two server clones in a blue/green deployment exercise. Removing the host from the inventory and resyncing fixes it until next time the server's public IP is flipped.