ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
14.05k stars 3.42k forks source link

Running template module in AWX 10.0.0 produces different results from AWX8.0.0 when using the same virtual environment #6536

Closed grahamn-gr closed 4 years ago

grahamn-gr commented 4 years ago
ISSUE TYPE
SUMMARY

When using AWX 10.0.0 to call the Ansible template module AWX 10.0.0 produces different results than AWX 8.0.0. This is when using the same virtual environment. When using the template module against a output file that already exists AWX 10.0.0 will report the output file has changed, when doing this against AWX 8.0.0 the file isn't changed.

It seems that AWX 10.0.0 calls the template module with different module_args than AWX 8.0.0 even though it's using the same virtual environment and Ansible version.

This can't be replicated in AWX 10.0.0 when running the playbook manually within the container via command line.

ENVIRONMENT
STEPS TO REPRODUCE

Have a git repo where there are already existing output files from the template module.

Setup AWX 8.0.0 with a virtualenv and point to it to a playbook that performs a template function that already produced the output files from above. There will be no changes.

Setup AWX 10.0.0 with the same virtualenv and point to it to a playbook that performs a template function that already produced the output files from above. There will be changes.

EXPECTED RESULTS

No changes detected

ACTUAL RESULTS

Changes detected

ADDITIONAL INFORMATION

When checking the invocation for the template module the module_args are different although we're using the same virtualenv which has the same Ansible version. I wouldn't expect these to change. The diff is also different too, file names have changed too.

AWX8.0.0

{
    "_ansible_parsed": true,
    "group": "root",
    "uid": 1000,
    "checksum": "f4ec7c23594121a180c1feaa2b669b2a7fcbfe60",
    "changed": false,
    "state": "file",
    "gid": 0,
    "mode": "0644",
    "invocation": {
        "module_args": {
            "directory_mode": null,
            "force": false,
            "remote_src": null,
            "path": "output/SITE/myconfig.cfg",
            "owner": null,
            "follow": false,
            "group": null,
            "unsafe_writes": null,
            "state": "file",
            "content": null,
            "serole": null,
            "diff_peek": null,
            "setype": null,
            "dest": "output/SITE/myconfig.cfg",
            "selevel": null,
            "original_basename": "myjinja.j2",
            "regexp": null,
            "validate": null,
            "src": "myjinja.j2",
            "seuser": null,
            "recurse": false,
            "delimiter": null,
            "mode": null,
            "attributes": null,
            "backup": null
        }
    },
    "owner": "awx",
    "diff": {
        "after": {
            "path": "output/SITE/myconfig.cfg"
        },
        "before": {
            "path": "output/SITE/myconfig.cfg"
        }
    },
    "path": "output/SITE/myconfig.cfg",
    "size": 45817,
    "_ansible_no_log": false
}

AWX 10.0.0

{
    "_ansible_parsed": true,
    "src": "/var/lib/awx/.ansible/tmp/ansible-tmp-1585821354.32-194655161154611/source",
    "md5sum": "6713681ba2a89120b255d81db6921d31",
    "group": "root",
    "uid": 1000,
    "dest": "output/SITE/myconfig.cfg",
    "checksum": "9c080b0fc34238dd8388296db38c7bc2633df1c2",
    "changed": true,
    "owner": "awx",
    "state": "file",
    "gid": 0,
    "mode": "0644",
    "invocation": {
        "module_args": {
            "directory_mode": null,
            "force": true,
            "remote_src": null,
            "owner": null,
            "follow": false,
            "local_follow": null,
            "group": null,
            "unsafe_writes": null,
            "setype": null,
            "content": null,
            "serole": null,
            "dest": "output/SITE/myconfig.cfg",
            "selevel": null,
            "original_basename": "myjinja.j2",
            "regexp": null,
            "validate": null,
            "src": "/var/lib/awx/.ansible/tmp/ansible-tmp-1585821354.32-194655161154611/source",
            "seuser": null,
            "delimiter": null,
            "mode": null,
            "attributes": null,
            "backup": false
        }
    },
    "diff": [
        {
            "after": "a lot of output",
            "after_header": "/tmp/tmpLSJWJr/myjinja.j2",
            "before_header": "output/SITE/myconfig.cfg",
            "before": "a lot of output but with some extra whitespace"
        }
    ],
    "size": 45817,
    "_ansible_no_log": false
}
ryanpetrello commented 4 years ago

@grahamn-gr,

Are you certain you're using the same Ansible version in both environments? This data structure comes from Ansible itself, so if it's changing in a major way, that suggests to me that the Ansible versions aren't the same.

grahamn-gr commented 4 years ago

Hi @ryanpetrello,

I'm sure. I'm using virtualenv "gr_awx_ansible_2_4_3_0" on both instances. I have a prod instance running now with 8.0.0 and a uat version running 10.0.0. I've extracted the pip list when in the virtualenv and the Ansible versions are the same.

I've pointed my job template to use the virtualenv on both instances however with 10.0.0 I see differences to templated files.

AWX 8.0.0 instance:

[grahamn@host Ansible-Networks]$ kubectl -n networks exec -it awx-0 -c awx-celery /bin/bash

(gr_awx_ansible_2_4_3_0) bash-4.2$ cat /var/lib/awx/.tower_version
8.0.0.0

bash-4.2$ ls -lrth /var/lib/awx/venv/
total 20K
drwxr-xr-x. 5 0 root 4.0K Oct 21 17:45 ansible
drwxr-xr-x. 1 0 root 4.0K Oct 21 17:46 awx
drwxr-xr-x. 7 0 root 4.0K Dec  2 16:47 gr_awx_ansible_2_9_0
drwxr-xr-x. 7 0 root 4.0K Dec  2 16:54 gr_awx_ansible_2_7_6
drwxr-xr-x. 7 0 root 4.0K Dec  2 16:57 gr_awx_ansible_2_4_3_0

bash-4.2$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.2$

(gr_awx_ansible_2_4_3_0) bash-4.2$ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                      Version
---------------------------- ---------
ansible                      2.4.3.0
ansible-tower-cli            3.3.0
asn1crypto                   1.2.0
Babel                        0.9.6
backports.ssl-match-hostname 3.4.0.2
bcrypt                       3.1.7
certifi                      2018.4.16
cffi                         1.11.5
chardet                      3.0.4
click                        6.7
colorama                     0.3.9
cryptography                 2.2.2
Cython                       0.29.10
dnspython                    1.12.0
enum34                       1.1.6
filelock                     2.0.6
gtextfsm                     0.2.1
hier-config                  1.5.1
idna                         2.6
iniparse                     0.4
ipaddr                       2.2.0
ipaddress                    1.0.22
Jinja2                       2.8
jmespath                     0.9.0
kitchen                      1.1.1
MarkupSafe                   0.11
netaddr                      0.7.19
ntlm-auth                    1.4.0
numpy                        1.16.4
pandas                       0.17.1
paramiko                     2.4.0
pexpect                      4.2.1
pip                          19.3.1
ply                          3.4
prettytable                  0.7.2
psutil                       5.4.3
psycopg2                     2.7.5
ptyprocess                   0.5
pyasn1                       0.4.2
pycparser                    2.18
pycrypto                     2.6.1
pycryptodomex                3.9.4
pycurl                       7.19.0
pyliblzma                    0.5.3
PyNaCl                       1.3.0
pysmi                        0.3.4
pysnmp                       4.4.4
python-dateutil              1.5
python-memcached             1.59
pytz                         2019.3
PyYAML                       3.12
requests                     2.19.1
requests-ntlm                1.1.0
setuptools                   42.0.2
six                          1.11.0
terminal                     0.4.0
urlgrabber                   3.10.2
urllib3                      1.23
wheel                        0.33.6

AWX 10.0.0 instance:

[grahamn@host Ansible-Networks]$ kubectl -n networks-uat exec -it awx-0 -c awx-task /bin/bash

(gr_awx_ansible_2_4_3_0) bash-4.4$  cat /var/lib/awx/.tower_version
10.0.0

bash-4.4$ ls -lrth /var/lib/awx/venv/
total 20K
drwxr-xr-x. 1 root root 4.0K Mar 30 17:22 awx
drwxr-xr-x. 6 root root 4.0K Mar 30 17:23 ansible
drwxr-xr-x. 8 root root 4.0K Apr  1 13:21 gr_awx_ansible_2_9_0
drwxr-xr-x. 8 root root 4.0K Apr  1 13:26 gr_awx_ansible_2_7_6
drwxr-xr-x. 8 root root 4.0K Apr  1 13:29 gr_awx_ansible_2_4_3_0

bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.4$

(gr_awx_ansible_2_4_3_0) bash-4.4$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/home/awx/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Package                      Version
---------------------------- ---------
ansible                      2.4.3.0
ansible-tower-cli            3.3.0
asn1crypto                   1.3.0
Babel                        0.9.6
backports.ssl-match-hostname 3.4.0.2
bcrypt                       3.1.7
certifi                      2018.4.16
cffi                         1.11.5
chardet                      3.0.4
click                        6.7
colorama                     0.3.9
cryptography                 2.2.2
Cython                       0.29.10
dnspython                    1.12.0
enum34                       1.1.6
filelock                     2.0.6
gtextfsm                     0.2.1
hier-config                  1.5.1
idna                         2.6
iniparse                     0.4
ipaddr                       2.2.0
ipaddress                    1.0.22
Jinja2                       2.8
jmespath                     0.9.0
kitchen                      1.1.1
MarkupSafe                   0.11
netaddr                      0.7.19
ntlm-auth                    1.4.0
numpy                        1.16.4
pandas                       0.17.1
paramiko                     2.4.0
pexpect                      4.2.1
pip                          20.0.2
ply                          3.4
prettytable                  0.7.2
psutil                       5.4.3
psycopg2                     2.7.5
ptyprocess                   0.5
pyasn1                       0.4.2
pycparser                    2.18
pycrypto                     2.6.1
pycryptodomex                3.9.7
pycurl                       7.19.0
pyliblzma                    0.5.3
PyNaCl                       1.3.0
pyodbc                       4.0.30
pysmi                        0.3.4
pysnmp                       4.4.4
python-dateutil              1.5
python-memcached             1.59
pytz                         2019.3
PyYAML                       3.12
requests                     2.19.1
requests-ntlm                1.1.0
setuptools                   44.1.0
six                          1.11.0
terminal                     0.4.0
urlgrabber                   3.10.2
urllib3                      1.23
wheel                        0.34.2

I've confirmed I'm pointing to the right virtualenv in the template;

AWX8.0.0 - No changes

image

AWX10.0.0 - Changes to some files which shouldn't be the case

image

wenottingham commented 4 years ago

can you confirm via the events that your 10.0.0 case is actually running the ansible in the venv (and not the system one)?

grahamn-gr commented 4 years ago

From my 10.0.0 environment if I turn on the most verbose logging I see this:

image

for the actual task that calls the template module I'm seeing this:

TASK [switch_template : switch_template | Generate configuration files for Switches] ***
task path: /tmp/awx_329_ro0wc7pl/project/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020  19:40:55 +0000 (0:00:00.021)       0:00:01.088 ******** 
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> PUT /tmp/tmp2wwfQH TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764 `" && echo ansible-tmp-1585856461.82-183887449063764="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764 `" ) && sleep 0'
<SWITCH> PUT /tmp/tmp_BPbOz TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/net_tools/basics/slurp.py
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> PUT /tmp/tmpaBQ13L TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856462.03-262270774623722/slurp.py
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856462.03-262270774623722/slurp.py && sleep 0'
<SWITCH> PUT /tmp/tmptQ6ah5/myjinja.j2 TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/source
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/source && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/copy.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/copy.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/copy.py; rm -rf "/var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/" > /dev/null 2>&1 && sleep 0'

Compare this to the AWX 8.0.0 host:

image

TASK [switch_template : switch_template | Generate configuration files for Switches] ***
task path: /tmp/awx_143069_mt6u2st6/project/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020  19:52:44 +0000 (0:00:00.023)       0:00:01.135 ******** 
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120 `" && echo ansible-tmp-1585857171.37-112353900880120="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120 `" ) && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/stat.py
<SWITCH> PUT /tmp/tmpoWPMpX TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> PUT /tmp/tmpYc4ecN TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py; rm -rf "/var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/" > /dev/null 2>&1 && sleep 0'

There's a lot more tasks going on with AWX 10.0.0 with mention of slurp.py and a PUT on the jinja template that doesn't occur with AWX 8.0.0.

I'll also note that I had to create a symlink for /usr/bin/python in order to continue to use our python2 virtualenv

ln --symbolic /usr/bin/python2 /usr/bin/python

This was needed in order resolve issues such as:

image

This is the task;

- name: create output directory
  run_once: yes
  delegate_to: localhost
  file: path={{ output_dir }} state=directory

If I run this via CLI using AWX 10.0.0 whilst in the virtualenv I don't see the same things as when AWX does the task:

Start of run:

bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.4$

(gr_awx_ansible_2_4_3_0) bash-4.4$ cat /var/lib/awx/.tower_version
10.0.0

(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook --version
ansible-playbook 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
  executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
  python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

(gr_awx_ansible_2_4_3_0) bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate

(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook --version
ansible-playbook 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
  executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
  python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

(gr_awx_ansible_2_4_3_0) bash-4.4$ cd /var/lib/awx/projects/_9__jenkins

(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook template.yml -i inventory/SWITCH/hosts --limit SWITCH --ask-vault-pass -vvvv
ansible-playbook 2.4.3.0
  config file = /var/lib/awx/projects/_9__jenkins/ansible.cfg
  configured module search path = [u'/var/lib/awx/projects/_9__jenkins/library']
  ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
  executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
  python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /var/lib/awx/projects/_9__jenkins/ansible.cfg as config file
Vault password:
setting up inventory plugins
Parsed /var/lib/awx/projects/_9__jenkins/inventory/EQLD/hosts inventory source with ini plugin
loading: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib64/python2.7/site-packages/ipaddr.py
loading: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ipaddr.py
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7f721a2ee090> for vault_id=default
Read vars_file 'network-vault.yaml'
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Read vars_file 'network-vault.yaml'
[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use 'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions. This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this rationale.. This feature will be removed in a
future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Read vars_file 'network-vault.yaml'
statically imported: /var/lib/awx/projects/_9__jenkins/roles/colo_trading_template/tasks/get_exchange_data.yml
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Loading callback plugin skippy of type stdout, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/skippy.pyc
Loading callback plugin profile_tasks of type aggregate, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/profile_tasks.pyc
Loading callback plugin timer of type aggregate, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/timer.pyc

Actual task:

TASK [switch_template : switch_template | Generate configuration files for Switches] *****************************************************************************************************************************
task path: /var/lib/awx/projects/_9__jenkins/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020  20:16:13 +0000 (0:00:00.019)       0:00:01.547 ********
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845 `" && echo ansible-tmp-1585858579.35-233657079709845="` echo /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845 `" ) && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/stat.py
<SWITCH> PUT /tmp/tmpPE22Xn TO /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/ /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> PUT /tmp/tmp0dqhqe TO /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/ /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py; rm -rf "/home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/" > /dev/null 2>&1 && sleep 0'
ok: [SWITCH] => {
    "changed": false,
    "checksum": "4efd9bd1c78b4bff7f522098264bf48631839579",
    "diff": {
        "after": {
            "path": "output/SITE/SWITCH.cfg"
        },
        "before": {
            "path": "output/SITE/SWITCH.cfg"
        }
    },
grahamn-gr commented 4 years ago

This is now resolved. Two things resolved it, I had to add a "sort" filter to one of my dicts. The other was to add the extra var to get ansible to use the right virtualenv. I can then see things are running with the correct virtualenv ansible_python_interpreter: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/python