search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.5k stars 3.34k forks source link

Stop pre-caching every resource in the system upon import #15128

Closed jbradberry closed 3 weeks ago

jbradberry commented 3 weeks ago
SUMMARY

The awx import command can take a long time before importing any objects, if the user has a lot of objects already in the database. This PR changes the import command to stop pre-loading objects, and instead do a targeted search when the cache fails to have a matching object.

ISSUE TYPE
COMPONENT NAME
jbradberry commented 3 weeks ago

In order to test, I created 10000 dummy users (this may have been overkill), then imported 71 existing users using both versions:

In [1]: for x in range(10000):
   ...:     User.objects.create_user(f"bob{x:04d}", f"bob{x:04d}@example.com", '
   ...: lkjasd;flkja;lsdkjflksj')
   ...:

old-style:

$ git checkout devel
$ time awx import -vv 2> jenkins-job-import8.txt < old-engineering-users.json

real    37m27.151s
user    35m42.804s
sys 0m14.702s

This didn't even succeed; it crashed after it dropped the connection and attempted to reset:

DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:https://localhost:8043 "OPTIONS /api/v2/users/ HTTP/1.1" 401 107
DEBUG:awxkit.api.client:"OPTIONS https://localhost:8043/api/v2/users/" elapsed: 0:00:00.017821
ERROR:awxkit.api.pages.page:This endpoint raised an error: /api/v2/users/
Traceback (most recent call last):
  File "/home/jrb/ansible/awx/awxkit/awxkit/cli/__init__.py", line 25, in run
    cli.parse_resource()
  File "/home/jrb/ansible/awx/awxkit/awxkit/cli/client.py", line 152, in parse_resource
    self.resource = parse_resource(self, skip_deprecated=skip_deprecated)
  File "/home/jrb/ansible/awx/awxkit/awxkit/cli/resource.py", line 222, in parse_resource
    response = command.handle(client, parser)
  File "/home/jrb/ansible/awx/awxkit/awxkit/cli/resource.py", line 145, in handle
    client.v2.import_assets(data)
  File "/home/jrb/ansible/awx/awxkit/awxkit/api/pages/api.py", line 425, in import_assets
    imported = self._import_list(endpoint, data.get(resource) or [])
  File "/home/jrb/ansible/awx/awxkit/awxkit/api/pages/api.py", line 277, in _import_list
    if field not in post_fields:
TypeError: argument of type 'NoneType' is not iterable

new-style:

$ git checkout cli-import-no-precache
$ time awx import -vv 2> jenkins-job-import9.txt < old-engineering-users.json

real    0m21.283s
user    0m1.794s
sys 0m0.084s

This one appears to have succeeded.