Closed grahamn-gr closed 4 years ago
@grahamn-gr,
Are you certain you're using the same Ansible version in both environments? This data structure comes from Ansible itself, so if it's changing in a major way, that suggests to me that the Ansible versions aren't the same.
Hi @ryanpetrello,
I'm sure. I'm using virtualenv "gr_awx_ansible_2_4_3_0" on both instances. I have a prod instance running now with 8.0.0 and a uat version running 10.0.0. I've extracted the pip list when in the virtualenv and the Ansible versions are the same.
I've pointed my job template to use the virtualenv on both instances however with 10.0.0 I see differences to templated files.
AWX 8.0.0 instance:
[grahamn@host Ansible-Networks]$ kubectl -n networks exec -it awx-0 -c awx-celery /bin/bash
(gr_awx_ansible_2_4_3_0) bash-4.2$ cat /var/lib/awx/.tower_version
8.0.0.0
bash-4.2$ ls -lrth /var/lib/awx/venv/
total 20K
drwxr-xr-x. 5 0 root 4.0K Oct 21 17:45 ansible
drwxr-xr-x. 1 0 root 4.0K Oct 21 17:46 awx
drwxr-xr-x. 7 0 root 4.0K Dec 2 16:47 gr_awx_ansible_2_9_0
drwxr-xr-x. 7 0 root 4.0K Dec 2 16:54 gr_awx_ansible_2_7_6
drwxr-xr-x. 7 0 root 4.0K Dec 2 16:57 gr_awx_ansible_2_4_3_0
bash-4.2$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.2$
(gr_awx_ansible_2_4_3_0) bash-4.2$ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------------------------- ---------
ansible 2.4.3.0
ansible-tower-cli 3.3.0
asn1crypto 1.2.0
Babel 0.9.6
backports.ssl-match-hostname 3.4.0.2
bcrypt 3.1.7
certifi 2018.4.16
cffi 1.11.5
chardet 3.0.4
click 6.7
colorama 0.3.9
cryptography 2.2.2
Cython 0.29.10
dnspython 1.12.0
enum34 1.1.6
filelock 2.0.6
gtextfsm 0.2.1
hier-config 1.5.1
idna 2.6
iniparse 0.4
ipaddr 2.2.0
ipaddress 1.0.22
Jinja2 2.8
jmespath 0.9.0
kitchen 1.1.1
MarkupSafe 0.11
netaddr 0.7.19
ntlm-auth 1.4.0
numpy 1.16.4
pandas 0.17.1
paramiko 2.4.0
pexpect 4.2.1
pip 19.3.1
ply 3.4
prettytable 0.7.2
psutil 5.4.3
psycopg2 2.7.5
ptyprocess 0.5
pyasn1 0.4.2
pycparser 2.18
pycrypto 2.6.1
pycryptodomex 3.9.4
pycurl 7.19.0
pyliblzma 0.5.3
PyNaCl 1.3.0
pysmi 0.3.4
pysnmp 4.4.4
python-dateutil 1.5
python-memcached 1.59
pytz 2019.3
PyYAML 3.12
requests 2.19.1
requests-ntlm 1.1.0
setuptools 42.0.2
six 1.11.0
terminal 0.4.0
urlgrabber 3.10.2
urllib3 1.23
wheel 0.33.6
AWX 10.0.0 instance:
[grahamn@host Ansible-Networks]$ kubectl -n networks-uat exec -it awx-0 -c awx-task /bin/bash
(gr_awx_ansible_2_4_3_0) bash-4.4$ cat /var/lib/awx/.tower_version
10.0.0
bash-4.4$ ls -lrth /var/lib/awx/venv/
total 20K
drwxr-xr-x. 1 root root 4.0K Mar 30 17:22 awx
drwxr-xr-x. 6 root root 4.0K Mar 30 17:23 ansible
drwxr-xr-x. 8 root root 4.0K Apr 1 13:21 gr_awx_ansible_2_9_0
drwxr-xr-x. 8 root root 4.0K Apr 1 13:26 gr_awx_ansible_2_7_6
drwxr-xr-x. 8 root root 4.0K Apr 1 13:29 gr_awx_ansible_2_4_3_0
bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.4$
(gr_awx_ansible_2_4_3_0) bash-4.4$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/home/awx/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Package Version
---------------------------- ---------
ansible 2.4.3.0
ansible-tower-cli 3.3.0
asn1crypto 1.3.0
Babel 0.9.6
backports.ssl-match-hostname 3.4.0.2
bcrypt 3.1.7
certifi 2018.4.16
cffi 1.11.5
chardet 3.0.4
click 6.7
colorama 0.3.9
cryptography 2.2.2
Cython 0.29.10
dnspython 1.12.0
enum34 1.1.6
filelock 2.0.6
gtextfsm 0.2.1
hier-config 1.5.1
idna 2.6
iniparse 0.4
ipaddr 2.2.0
ipaddress 1.0.22
Jinja2 2.8
jmespath 0.9.0
kitchen 1.1.1
MarkupSafe 0.11
netaddr 0.7.19
ntlm-auth 1.4.0
numpy 1.16.4
pandas 0.17.1
paramiko 2.4.0
pexpect 4.2.1
pip 20.0.2
ply 3.4
prettytable 0.7.2
psutil 5.4.3
psycopg2 2.7.5
ptyprocess 0.5
pyasn1 0.4.2
pycparser 2.18
pycrypto 2.6.1
pycryptodomex 3.9.7
pycurl 7.19.0
pyliblzma 0.5.3
PyNaCl 1.3.0
pyodbc 4.0.30
pysmi 0.3.4
pysnmp 4.4.4
python-dateutil 1.5
python-memcached 1.59
pytz 2019.3
PyYAML 3.12
requests 2.19.1
requests-ntlm 1.1.0
setuptools 44.1.0
six 1.11.0
terminal 0.4.0
urlgrabber 3.10.2
urllib3 1.23
wheel 0.34.2
I've confirmed I'm pointing to the right virtualenv in the template;
AWX8.0.0 - No changes
AWX10.0.0 - Changes to some files which shouldn't be the case
can you confirm via the events that your 10.0.0 case is actually running the ansible in the venv (and not the system one)?
From my 10.0.0 environment if I turn on the most verbose logging I see this:
for the actual task that calls the template module I'm seeing this:
TASK [switch_template : switch_template | Generate configuration files for Switches] ***
task path: /tmp/awx_329_ro0wc7pl/project/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020 19:40:55 +0000 (0:00:00.021) 0:00:01.088 ********
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> PUT /tmp/tmp2wwfQH TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764 `" && echo ansible-tmp-1585856461.82-183887449063764="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764 `" ) && sleep 0'
<SWITCH> PUT /tmp/tmp_BPbOz TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.82-183887449063764/file.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/net_tools/basics/slurp.py
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> PUT /tmp/tmpaBQ13L TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856462.03-262270774623722/slurp.py
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856462.03-262270774623722/slurp.py && sleep 0'
<SWITCH> PUT /tmp/tmptQ6ah5/myjinja.j2 TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/source
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/source && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/copy.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/copy.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/copy.py; rm -rf "/var/lib/awx/.ansible/tmp/ansible-tmp-1585856461.51-24327124823313/" > /dev/null 2>&1 && sleep 0'
Compare this to the AWX 8.0.0 host:
TASK [switch_template : switch_template | Generate configuration files for Switches] ***
task path: /tmp/awx_143069_mt6u2st6/project/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020 19:52:44 +0000 (0:00:00.023) 0:00:01.135 ********
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120 `" && echo ansible-tmp-1585857171.37-112353900880120="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120 `" ) && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/stat.py
<SWITCH> PUT /tmp/tmpoWPMpX TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> PUT /tmp/tmpYc4ecN TO /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/ /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/file.py; rm -rf "/var/lib/awx/.ansible/tmp/ansible-tmp-1585857171.37-112353900880120/" > /dev/null 2>&1 && sleep 0'
There's a lot more tasks going on with AWX 10.0.0 with mention of slurp.py and a PUT on the jinja template that doesn't occur with AWX 8.0.0.
I'll also note that I had to create a symlink for /usr/bin/python in order to continue to use our python2 virtualenv
ln --symbolic /usr/bin/python2 /usr/bin/python
This was needed in order resolve issues such as:
This is the task;
- name: create output directory
run_once: yes
delegate_to: localhost
file: path={{ output_dir }} state=directory
If I run this via CLI using AWX 10.0.0 whilst in the virtualenv I don't see the same things as when AWX does the task:
Start of run:
bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.4$
(gr_awx_ansible_2_4_3_0) bash-4.4$ cat /var/lib/awx/.tower_version
10.0.0
(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook --version
ansible-playbook 2.4.3.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
(gr_awx_ansible_2_4_3_0) bash-4.4$ source /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/activate
(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook --version
ansible-playbook 2.4.3.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
(gr_awx_ansible_2_4_3_0) bash-4.4$ cd /var/lib/awx/projects/_9__jenkins
(gr_awx_ansible_2_4_3_0) bash-4.4$ ansible-playbook template.yml -i inventory/SWITCH/hosts --limit SWITCH --ask-vault-pass -vvvv
ansible-playbook 2.4.3.0
config file = /var/lib/awx/projects/_9__jenkins/ansible.cfg
configured module search path = [u'/var/lib/awx/projects/_9__jenkins/library']
ansible python module location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible
executable location = /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/ansible-playbook
python version = 2.7.16 (default, Nov 17 2019, 00:07:27) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /var/lib/awx/projects/_9__jenkins/ansible.cfg as config file
Vault password:
setting up inventory plugins
Parsed /var/lib/awx/projects/_9__jenkins/inventory/EQLD/hosts inventory source with ini plugin
loading: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib64/python2.7/site-packages/ipaddr.py
loading: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ipaddr.py
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7f721a2ee090> for vault_id=default
Read vars_file 'network-vault.yaml'
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Read vars_file 'network-vault.yaml'
[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use 'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions. This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this rationale.. This feature will be removed in a
future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Read vars_file 'network-vault.yaml'
statically imported: /var/lib/awx/projects/_9__jenkins/roles/colo_trading_template/tasks/get_exchange_data.yml
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
skipping vars_file 'inventory/{{ site }}/group_vars/extra_vars.yml' due to an undefined variable
Loading callback plugin skippy of type stdout, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/skippy.pyc
Loading callback plugin profile_tasks of type aggregate, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/profile_tasks.pyc
Loading callback plugin timer of type aggregate, v2.0 from /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/plugins/callback/timer.pyc
Actual task:
TASK [switch_template : switch_template | Generate configuration files for Switches] *****************************************************************************************************************************
task path: /var/lib/awx/projects/_9__jenkins/roles/leaf_template/tasks/main.yml:51
Thursday 02 April 2020 20:16:13 +0000 (0:00:00.019) 0:00:01.547 ********
<SWITCH> ESTABLISH LOCAL CONNECTION FOR USER: awx
<SWITCH> EXEC /bin/sh -c 'echo ~ && sleep 0'
<SWITCH> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845 `" && echo ansible-tmp-1585858579.35-233657079709845="` echo /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845 `" ) && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/stat.py
<SWITCH> PUT /tmp/tmpPE22Xn TO /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/ /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/stat.py && sleep 0'
Using module file /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/lib/python2.7/site-packages/ansible/modules/files/file.py
<SWITCH> PUT /tmp/tmp0dqhqe TO /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py
<SWITCH> EXEC /bin/sh -c 'chmod u+x /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/ /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py && sleep 0'
<SWITCH> EXEC /bin/sh -c '/usr/bin/python /home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/file.py; rm -rf "/home/awx/.ansible/tmp/ansible-tmp-1585858579.35-233657079709845/" > /dev/null 2>&1 && sleep 0'
ok: [SWITCH] => {
"changed": false,
"checksum": "4efd9bd1c78b4bff7f522098264bf48631839579",
"diff": {
"after": {
"path": "output/SITE/SWITCH.cfg"
},
"before": {
"path": "output/SITE/SWITCH.cfg"
}
},
This is now resolved. Two things resolved it, I had to add a "sort" filter to one of my dicts. The other was to add the extra var to get ansible to use the right virtualenv. I can then see things are running with the correct virtualenv ansible_python_interpreter: /var/lib/awx/venv/gr_awx_ansible_2_4_3_0/bin/python
ISSUE TYPE
SUMMARY
When using AWX 10.0.0 to call the Ansible template module AWX 10.0.0 produces different results than AWX 8.0.0. This is when using the same virtual environment. When using the template module against a output file that already exists AWX 10.0.0 will report the output file has changed, when doing this against AWX 8.0.0 the file isn't changed.
It seems that AWX 10.0.0 calls the template module with different module_args than AWX 8.0.0 even though it's using the same virtual environment and Ansible version.
This can't be replicated in AWX 10.0.0 when running the playbook manually within the container via command line.
ENVIRONMENT
STEPS TO REPRODUCE
Have a git repo where there are already existing output files from the template module.
Setup AWX 8.0.0 with a virtualenv and point to it to a playbook that performs a template function that already produced the output files from above. There will be no changes.
Setup AWX 10.0.0 with the same virtualenv and point to it to a playbook that performs a template function that already produced the output files from above. There will be changes.
EXPECTED RESULTS
No changes detected
ACTUAL RESULTS
Changes detected
ADDITIONAL INFORMATION
When checking the invocation for the template module the module_args are different although we're using the same virtualenv which has the same Ansible version. I wouldn't expect these to change. The diff is also different too, file names have changed too.
AWX8.0.0
AWX 10.0.0