Add documention for using ARA with AWX/Tower

[dmsimard]

ARA and AWX/Tower aren't mutually exclusive. They can be used individually or together depending on the user's needs and use cases.

In theory, we would be able to install ara in the default virtualenv but that doesn't work and is documented in https://github.com/ansible/awx/issues/1737

What had worked for me in the past would be to create a new virtualenv, install ara and ansible in it and then make awx/tower use that virtualenv for job templates after setting the callback plugin in the settings interface.

Screencaps from an Ansible meetup presentation:

...

[6sossomons]

I've used those same steps as described above and still don't see data being recorded into ara from runs inside tower.

[dmsimard]

@6sossomons yes, at the time I created the issue I also noted that the procedure didn't work and needed to be troubleshooted.

I got it to work just now:

And in ARA:

The issue was that job isolation (in settings -> jobs) was enabled and /var/lib/awx/.ara was not in the list of "paths exposed to isolated jobs".

Considering that AWX/Tower requires postgresql, it might be wise to consider documenting setting it up in postgresql instead of sqlite anyway.

[MrMEEE]

Also trying to get it to work here..

I have configured the following:

But nothing is pushed into the database.. it works fine if I run it from the cli...

Anyone got a hint??... anyway to troubleshoot/debug this

[dmsimard]

@MrMEEE you can see this in my first comments but when I got it to work, it was by creating the venv in /var/lib/awx/venv/ara. This is somehow picked up by awx automatically and the venv becomes available in a dropdown menu to select in your job templates.

[MrMEEE]

hi @dmsimard

I tried that as well.. still does nothing :(

[6sossomons]

Any update on this case?

[dmsimard]

@6sossomons @MrMEEE I haven't had the opportunity to look into this yet, been focused on releasing ara 1.0.

In other news, Tower 3.5 has been released for RHEL8 with full python3 support which will make it easier to integrate ara 1.0.

In the meantime, ara 1.0 is already integration tested with postgresql so we could have ara leverage the AWX/Tower postgresql instance for data storage.

The release of CentOS 8 is also making progress: https://wiki.centos.org/About/Building_8

I won't have time to look at this for a while still but I'd appreciate any help.

[HOSTED-POWER]

It would be great if there is some documentation. We use awx in docker but weren't able to get ARA working either unfortunately :(

[dmsimard]

@HOSTED-POWER would you like to share the steps you tried and what didn't work ?

[dmsimard]

I haven't had time to spend on this yet but I came across a workaround that someone posted on StackOverflow and thought it would be worth adding to this issue: https://stackoverflow.com/a/56728592/6072068

[6sossomons]

I tried the work-around with no avail. All of my venv are running ara in them, so now I just need to get ara to play nicely inside the scope of tower.

[dmsimard]

No progress on this yet but cross-referencing a reply I did to a post on /r/ansible about integration with AWX/Tower: https://www.reddit.com/r/ansible/comments/eo6k4x/ansible_ara_integration_help/

[dmsimard]

I spent some time on this last night and got it working on the latest version of AWX on CentOS 8.

In a nutshell:

# Install ARA in the awx_task container
docker exec -it awx_task /var/lib/awx/venv/ansible/bin/pip install ara
Requirement already satisfied: ara in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (1.3.2)
Requirement already satisfied: requests>=2.14.2 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from ara) (2.22.0)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from ara) (5.4.4)
Requirement already satisfied: idna<2.9,>=2.5 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from requests>=2.14.2->ara) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from requests>=2.14.2->ara) (2019.11.28)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from requests>=2.14.2->ara) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /var/lib/awx/venv/ansible/lib/python3.6/site-packages (from requests>=2.14.2->ara) (1.25.7)

# Get path to callback plugins so we can configure it later
docker exec -it awx_task /var/lib/awx/venv/ansible/bin/python3 -m ara.setup.callback_plugins
/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback

# Install and launch ARA API server on the host in the background (not in a container)
python3 -m pip install ara[server] --user
ara-manage runserver 0.0.0.0:8000 &

Some screen captures:

Setting up the callback plugins path and environment variables for configuring the ARA callback

Note: For the ARA_API_SERVER variable, using 127.0.0.1 didn't work and I had to specify the local IP address (10.0.0.9 in my case). This meant I had to add 10.0.0.9 to the list of ALLOWED_HOSTS in ~/.ara/server/settings.yaml.

Running the included demo template

Playbook task result

Playbook task file

Playbook extra_vars file

Host facts

[dmsimard]

Following up on my previous comment, I've created a proof of concept gist that takes care of installing both AWX and ARA with instructions on how to setup AWX to send data to ARA: https://gist.github.com/dmsimard/368fa69e3927001f65a1b0d56fee7f02

It is minimal but a good starting point that demonstrates the integration working.

In terms of next steps, I think we need the following to do this cleanly:

• Add support for CentOS 8 for the ara_api and ara_frontend Ansible roles. I've created this issue for tracking this: https://github.com/ansible-community/ara/issues/107
• There is also an issue for adding support to ara_api so it can install ARA with pip install --user instead of inside a virtualenv: https://github.com/ansible-community/ara/issues/82. I don't think this is a blocker, though, as the API server can run from a virtualenv just as well.

Afterwards, it would probably make sense to write a small playbook that would create a postgresql database on the AWX postgresql instance so ARA can use that instead of the sqlite default.

[VishMudemela]

@dmsimard i followed above instructions and install ara in my awx, but i am not seeing any results in ara, when i ran playbook i am getting below:

` ansible-playbook 2.9.14 config file = /etc/ansible/ansible.cfg configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.6/site-packages/ansible executable location = /usr/bin/ansible-playbook python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] Using /etc/ansible/ansible.cfg as config file setting up inventory plugins host_list declined parsing /tmp/awx_311_nk7jkg7i/tmpegj447bg as it did not pass its verify_file() method Set default localhost to localhost Parsed /tmp/awx_311_nk7jkg7i/tmpegj447bg inventory source with script plugin Loading callback plugin awx_display of type stdout, v2.0 from /var/lib/awx/venv/awx/lib/python3.6/site-packages/ansible_runner/callbacks/awx_display.py Loading callback plugin ara_default of type awesome, v2.0 from /var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py

PLAYBOOK: hello_world.yml ** Positional arguments: hello_world.yml verbosity: 4 remote_user: admin connection: smart timeout: 10 become_method: sudo tags: ('all',) inventory: ('/tmp/awx_311_nk7jkg7i/tmpegj447bg',) extra_vars: ('@/tmp/awx_311_nk7jkg7i/env/extravars',) forks: 5 [WARNING]: Failure using method (v2_playbook_on_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f33ab7bdc50>): HTTPConnectionPool(host='10.215.20.14', port=80): Max retries exceeded with url: /api/v1/playbooks (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f33aac0a3c8>: Failed to establish a new connection: [Errno 111] Connection refused',)) Callback Exception: File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback method(*new_args, kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 274, in v2_playbook_on_start started=datetime.datetime.now().isoformat(), File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/clients/http.py", line 115, in post return self._request("post", endpoint, kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/clients/http.py", line 93, in _request response = func(url, kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/clients/http.py", line 69, in post return self._request("post", url, data=json.dumps(payload)) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/clients/http.py", line 57, in _request return self.http.request(method, self.endpoint + url, timeout=self.timeout, payload) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) 1 plays in hello_world.yml

PLAY [Hello World Sample] ** [WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f33ab7bdc50>): 'NoneType' object is not subscriptable Callback Exception: File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback method(*new_args, *kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 304, in v2_playbook_on_play_start self._set_playbook_labels(labels=labels) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 428, in _set_playbook_labels current_labels = [label["name"] for label in self.playbook["labels"]] [WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f33ab7bdc50>): 'NoneType' object is not subscriptable Callback Exception: File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback method(new_args, **kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 342, in v2_playbook_on_task_start task_file = self._get_or_create_file(path) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 450, in _get_or_create_file "/api/v1/files", playbook=self.playbook["id"], path=path, content=content

ESTABLISH LOCAL CONNECTION FOR USER: root EXEC /bin/sh -c 'echo ~root && sleep 0' EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331 `" && echo ansible-tmp-1607548197.246158-24431-175708366360331="` echo /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331 `" ) && sleep 0' Using module file /usr/lib/python3.6/site-packages/ansible/modules/system/setup.py PUT /var/lib/awx/.ansible/tmp/ansible-local-24424e7qp2yof/tmpdwjw04rd TO /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331/AnsiballZ_setup.py EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331/ /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331/AnsiballZ_setup.py && sleep 0' EXEC /bin/sh -c '/usr/bin/python3.6 /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331/AnsiballZ_setup.py && sleep 0' EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1607548197.246158-24431-175708366360331/ > /dev/null 2>&1 && sleep 0' TASK [Gathering Facts] ********************************************************* task path: /tmp/awx_311_nk7jkg7i/project/hello_world.yml:1 ok: [localhost] META: ran handlers Callback Exception: File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback method(*new_args, **kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 342, in v2_playbook_on_task_start task_file = self._get_or_create_file(path) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 450, in _get_or_create_file "/api/v1/files", playbook=self.playbook["id"], path=path, content=content TASK [Hello Message] *********************************************************** task path: /tmp/awx_311_nk7jkg7i/project/hello_world.yml:4 ok: [localhost] => { "msg": "Hello World!" } META: ran handlers META: ran handlers PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 [WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin (): 'NoneType' object is not subscriptable Callback Exception: File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback method(*new_args, **kwargs) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 380, in v2_playbook_on_stats self._load_stats(stats) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 514, in _load_stats host = self._get_or_create_host(hostname) File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/ara/plugins/callback/ara_default.py", line 459, in _get_or_create_host self.host_cache[host] = self.client.post("/api/v1/hosts", name=host, playbook=self.playbook["id"]) `

[VishMudemela]

working now, there was a typo in setting->jobs->

i forgot to give port 8000

[apinter]

Big thanks @dmsimard for the write up. I just deployed today AWX using the AWX-operator on openSUSE MicroOS in k3s and can already see that this is not going to be as smooth as I imagined. Is this the latest doc on the integration or is there something we need to keep an eye out for if we're using newer versions of AWX? Or I should stop worrying about it, do it and report back if I see something? ^_^

[dmsimard]

Hey @apinter, there are no formal documentation for integrating ara with awx yet.

I haven't revisited this issue in a while and haven't even attempted to try ara with the new k8s deployment model with the operator for awx using container images and execution environments (EEs) instead of virtual environments. If you manage to make it work with awx-operator, I am definitely interested to know how :)

I can point you in the right direction, though. Broadly speaking, the callback plugin needs to be installed and configured wherever the ansible commands are running from.

Before, this might have meant running something like docker exec -it awx_task /var/lib/awx/venv/ansible/bin/pip install ara but with the shift to container images (EEs), I suppose you would need to add a layer to the EE image where you install ara and then make the operator use this new image. There is a ee_images argument in the awx-operator that would probably allow you to use that new custom image once it is built and hosted somewhere.

You'll probably want to host a persistent API server somewhere so the different containers can send data to it. ara has container images on quay and dockerhub. Once the ara server is up at a known location, you'll need to supply the callback with the location of that server and credentials (if need be), I suppose that would be in the extra environment variables portion of AWX's settings like in this screenshot from a previous comment:

Let me know if you get stuck somewhere and I can try to help but I don't have time to work on this in the near future.

[apinter]

Thanks @dmsimard, going to start looking into this deeper today and possibly during the weekend. Anyhow the current setup is a single node k3s and the host has the API server installed and running as well (w/ pip), but I assume that if the modified container image is in place and works that could be replicated in a bigger cluster as well.

Will check back here if I have something or need help - which I might definitely need ^_^.

EDIT: Things got busy, but got to the point where I finally have a Kubic cluster ready and have day off so can start checking things out.

[dmsimard]

Hi everyone,

I just want to briefly report back -- I haven't got everything written down but do have a working proof of concept using the latest version of AWX (19.5.0) deployed with awx-operator 0.15.0.

At a high level:

• Deploy awx with the awx-operator
• Build (or get) a new EE that includes the ara package (I've have a proof of concept on quay.io that uses ansible-4.10.0)
• Add the EE into AWX
• Set the right callback plugin path as well as ara callback configuration environment variables in the AWX job settings
• Configure the job templates to use the execution environment image that contains the ara package
• Run as usual

I will take more time to flesh the instructions above when I have a chance but it should be a good start to point you in the right direction if you want to try it out.

Screenshots

Fresh install

Job settings

Custom execution environment

Running a job template

Recorded in ara

[dmsimard]

I've written a blog post on how to do it at a high level with the latest version of AWX (as of writing): https://ara.recordsansible.org/blog/2021/12/23/recording-ansible-playbooks-from-awx-with-ara/