No module named six

[tardoe]

Installing the Ansible pack with --python3 installs fine but no commands can be run as the install appears to think it's missing the module 'six'.

This might be somewhat related to https://github.com/StackStorm-Exchange/stackstorm-ansible/issues/5

root@ss01:/opt/stackstorm# st2 run ansible.command_local args="echo 'hi'"
..
id: 5c7d9592253fde2e09b23a24
status: failed
parameters:
  args: echo 'hi'
  cwd: /opt/stackstorm/packs/ansible
result:
  failed: true
  return_code: 1
  stderr: "Traceback (most recent call last):
  File "/opt/stackstorm/packs/ansible/actions/ansible.py", line 4, in <module>
    from lib.ansible_base import AnsibleBaseRunner
  File "/opt/stackstorm/packs/ansible/actions/lib/ansible_base.py", line 7, in <module>
    import six
ImportError: No module named six"
  stdout: ''
  succeeded: false

(ansible) root@ss01:/opt/stackstorm# ansible --version
ansible 2.7.8
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python3.5/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible
  python version = 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]

(ansible) root@ss01:/opt/stackstorm# pip list --format=columns
Package           Version
----------------- ----------
ansible           2.7.8
asn1crypto        0.24.0
bcrypt            3.1.6
certifi           2018.11.29
cffi              1.12.2
chardet           3.0.4
cryptography      2.6.1
idna              2.8
Jinja2            2.10
MarkupSafe        1.1.1
ntlm-auth         1.2.0
paramiko          2.4.2
pip               9.0.1
pyasn1            0.4.5
pycparser         2.19
pykerberos        1.2.1
PyNaCl            1.3.0
pyOpenSSL         19.0.0
pywinrm           0.3.0
PyYAML            3.13
requests          2.21.0
requests-credssp  1.0.2
requests-kerberos 0.12.0
requests-ntlm     1.1.0
setuptools        40.8.0
six               1.12.0
urllib3           1.24.1
wheel             0.29.0
xmltodict         0.12.0

[tardoe]

This looks to not only apply to py3...

root@ss01:/opt/stackstorm/virtualenvs/ansible# st2 run ansible.command_local args="echo $TERM"
.
id: 5c808637253fde6e72f50893
status: failed
parameters:
  args: echo xterm-256color
  cwd: /opt/stackstorm/packs/ansible
result:
  failed: true
  return_code: 1
  stderr: "Traceback (most recent call last):
  File "/opt/stackstorm/packs/ansible/actions/ansible.py", line 4, in <module>
    from lib.ansible_base import AnsibleBaseRunner
  File "/opt/stackstorm/packs/ansible/actions/lib/ansible_base.py", line 7, in <module>
    import six
ImportError: No module named six"
  stdout: ''
  succeeded: false

(ansible) root@ss01:/opt/stackstorm/virtualenvs/ansible# ansible --version
ansible 2.7.8
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

[arm4b]

Thanks for the report!

I couldn't reproduce this on latest ST2 (st2 2.10.2, on Python 2.7.12 under Ubuntu16) in both py2 and py3 environments:

Py3

vagrant@stackstorm:~$ /opt/stackstorm/virtualenvs/ansible/bin/ansible --version
ansible 2.7.8
  config file = None
  configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python3.5/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible
  python version = 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]

and

vagrant@stackstorm:~$ st2 run ansible.command_local args="echo $TERM"
.
id: 5c829f6b6cb8de05d876ae28
status: succeeded
parameters: 
  args: echo xterm-256color
  cwd: /opt/stackstorm/packs/ansible
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: '127.0.0.1 | CHANGED | rc=0 >>
    xterm-256color'
  succeeded: true

---

What's your ST2 version, how did you install it? Any modifications made to st2 core or configuration, - I guess there could be some diff here.

Worth trying to grab a fresh st2 demo box from https://docs.stackstorm.com/install/vagrant.html and try to run it again, search for the diff in environment.

[tardoe]

Brand new install using puppet-st2 on the latest version.

root@ss01:/opt/stackstorm# st2 run ansible.command_local args='ping 8.8.8.8'
.
id: 5c9c7e15253fde25a847f9f8
status: failed
parameters:
  args: ping 8.8.8.8
  cwd: /opt/stackstorm/packs/ansible
result:
  failed: true
  return_code: 1
  stderr: "Traceback (most recent call last):
  File "/opt/stackstorm/packs/ansible/actions/ansible.py", line 4, in <module>
    from lib.ansible_base import AnsibleBaseRunner
  File "/opt/stackstorm/packs/ansible/actions/lib/ansible_base.py", line 7, in <module>
    import six
ImportError: No module named six"
  stdout: ''
  succeeded: false

root@ss01:/opt/stackstorm# st2 --version
st2 2.10.4, on Python 2.7.12

[arm4b]

Try to find the chance to check this scenario in pre-packaged Vagrant demo box and see if it works there: https://docs.stackstorm.com/install/vagrant.html Wondering if there is the diff in environment or not.

[tardoe]

Okay, that worked in the Vagrant image - after installing build-essential and libkrb5-dev to install the ansible pack, it works correctly. Looks like this is down to the Puppet install?

After the puppet install completes, it passes all st2-self-check tests. Where to next?

[arm4b]

From one side it could be VM environment (OS/image/custom configuration/etc?), from the other, - installer. I'd try permutations to understand which one is guilty.

For example picking a clean & fresh Ubuntu Xenial VM (like we have in that base Vagrant) and install ST2 with most used scripted installer on top of it https://docs.stackstorm.com/install/index.html#ref-one-line-install vs Puppet installer. Compare if there is a diff.

vagrant init ubuntu/xenial64
vagrant up
vagrant ssh

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password='Ch@ngeMe'

[tardoe]

I deployed this into our standard SOE on 16.04 LTS (latest). We don’t do anything with python, venv etc as it’s all left to the puppet manifest.

I’ll attempt with a scripting install and a vanilla puppet-apply install. I’ll reach out to the Puppet module team as well in case they have any bright ideas.

Regards,

Tim Raphael

On 29 Mar 2019, at 08:54, Eugen C. notifications@github.com wrote:

From one side it could be VM environment (OS/image/custom configuration/etc?), from the other, - installer. I'd try permutations to understand which one is guilty.

For example picking a clean & fresh Ubuntu Xenial VM (like we have in that base Vagrant) and install ST2 with scripted installer on top of it https://docs.stackstorm.com/install/index.html#ref-one-line-install vs Puppet installer. Compare if there is a diff.

vagrant init ubuntu/xenial64 vagrant up vagrant ssh

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password='Ch@ngeMe' — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[arm4b]

Sure, let us know here how it goes and your findings. If it leads to catching a bug in any part of the deployment process, - that would be a great outcome.

[tardoe]

Okay, this looks like it's isolated to installation by Puppet.

curl | bash single line install method on a clean 16.04 full patched box is fine. Vagrant box install - fine.

puppet apply method fails exactly the same as my "Infrastructure" Puppet install.

[nmaludy]

Helping @tardoe dig into this on Slack and think we figured out what is going on.

The ansible.command_local action uses the local-shell-script runner: https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/master/actions/command_local.yaml#L4-L5

It then executes ansible.py using the system python and system PYTHONPATH, NOT the virtualenv of the ansible pack!

Since six isn't installed by default on Ubuntu16.04, the action fails.

To verify this i modified actions/ansible.py to print out the python executable being used and the PYTHONPATH and got the following:

python executable = /usr/bin/python
python path = [
    "/opt/stackstorm/packs/ansible/actions", 
    "/usr/lib/python2.7", 
    "/usr/lib/python2.7/plat-x86_64-linux-gnu", 
    "/usr/lib/python2.7/lib-tk", 
    "/usr/lib/python2.7/lib-old", 
    "/usr/lib/python2.7/lib-dynload", 
    "/usr/local/lib/python2.7/dist-packages", 
    "/usr/lib/python2.7/dist-packages"
]

@armab I think the best way to fix this is to NOT use local-shell-script runner and instead use the python-script runner for the actions. Not sure why local-shell-script runner was being used in the first place?

[arm4b]

Debugging statements in there are irrelevant as it comes from a base helper python script which just a wrapper and then forks subprocess /opt/stackstorm/virtualenvs/ansible/bin/ansible that is enough for switching to just installed virtualenv-ed Ansible: https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/30b29dd87fa0e765679bb3d3784bec2b69c48b16/actions/lib/ansible_base.py#L105

We don't run Ansible via its API, but executing it by binary and so python-script vs local-shell-script makes no difference here.

If you edit /opt/stackstorm/virtualenvs/bin/ansible to add similar debugging statements, it shows the following for me:

st2 run ansible.command connection=local inventory_file='127.0.0.1,' hosts=all args='echo $PATH'

result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: 'Ansible Executable: /opt/stackstorm/virtualenvs/ansible/bin/python
    Ansible Python path: [''/opt/stackstorm/virtualenvs/ansible/bin'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7/plat-x86_64-linux-gnu'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7/lib-tk'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7/lib-old'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7/lib-dynload'', ''/usr/lib/python2.7'', ''/usr/lib/python2.7/plat-x86_64-linux-gnu'', ''/usr/lib/python2.7/lib-tk'', ''/opt/stackstorm/virtualenvs/ansible/local/lib/python2.7/site-packages'', ''/opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages'']
    127.0.0.1 | CHANGED | rc=0 >>
    /opt/stackstorm/virtualenvs/ansible/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin'
  succeeded: true

---

@nmaludy I'm curious what kind of configuration was different when installing via Puppet vs default installation as there is no problem with six there? Let's try to find the diff.

[arm4b]

Figured out with @nmaludy what's going on there.

On some systems six python package is absent, which is utilized by the helper scripts here: https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/30b29dd87fa0e765679bb3d3784bec2b69c48b16/actions/lib/ansible_base.py#L7

curl|bash installer adds crudini which then pulls python-six as a dependency. That's why the diff we see in Puppet based install which manages st2.conf itself, but not in curl|bash installer which manages st2.conf by crudini.

$ sudo apt install crudini
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  python-iniparse python-six
The following NEW packages will be installed:
  crudini python-iniparse python-six
0 upgraded, 3 newly installed, 0 to remove and 71 not upgraded.
Need to get 42.9 kB of archives.
After this operation, 218 kB of additional disk space will be used.
Do you want to continue? [Y/n]

---

It's indeed bug in ansible pack.

Thanks @tardoe for reporting it and going back and forward with different environments and @nmaludy for catching the root cause :+1: