search

TerryHowe / ansible-modules-hashivault

Ansible module for Hashicorp Vault.
https://pypi.python.org/pypi/ansible-modules-hashivault
MIT License
454 stars 158 forks source link

No module named hvac after upgrading to ansible version 6.0.0 or 6.1.0 / module is running under python 2.7??! #412

Open df-cgdm opened 2 years ago

df-cgdm commented 2 years ago

After upgrading ansible from version 5.10.0 to version 6.0.0 or 6.1.0 I'm getting the error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: No module named hvac
fatal: [localhost]: FAILED! => {"changed": false, "failed_when_result": true, "module_stderr": "Traceback (most recent call last):\n  File \"/home/adm-dfournout/.ansible/tmp/ansible-tmp-1659342981.5344145-63417-27544398947102/AnsiballZ_hashivault_read.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/home/adm-dfournout/.ansible/tmp/ansible-tmp-1659342981.5344145-63417-27544398947102/AnsiballZ_hashivault_read.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/adm-dfournout/.ansible/tmp/ansible-tmp-1659342981.5344145-63417-27544398947102/AnsiballZ_hashivault_read.py\", line 48, in invoke_module\n    run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python2.7/runpy.py\", line 188, in run_module\n    fname, loader, pkg_name)\n  File \"/usr/lib/python2.7/runpy.py\", line 82, in _run_module_code\n    mod_name, mod_fname, mod_loader, pkg_name)\n  File \"/usr/lib/python2.7/runpy.py\", line 72, in _run_code\n    exec code in run_globals\n  File \"/tmp/ansible_ansible.legacy.hashivault_read_payload_tZdTZc/ansible_ansible.legacy.hashivault_read_payload.zip/ansible/modules/hashivault/hashivault_read.py\", line 3, in <module>\n  File \"/tmp/ansible_ansible.legacy.hashivault_read_payload_tZdTZc/ansible_ansible.legacy.hashivault_read_payload.zip/ansible/module_utils/hashivault.py\", line 3, in <module>\nImportError: No module named hvac\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

What I'm seeing is the module is running with "python 2.7" but it should run under "python 3.9" (as it was with version 5.10.0).

Does someone has already used this module with ansible 6 (ansible core 2.13) ? Is there any configuration to be done on ansible 6 to make it works ?

df-cgdm commented 2 years ago

The problem seems to come from the "shebang" in the modules. When I remove the shebang from "hashivault_read.py", it's working. Replacing the shebang "#!/usr/bin/env python" by "#!/usr/bin/python" is also working.

TerryHowe commented 2 years ago

What happens when you run /usr/bin/env python ? no such file or directory?

What distro are you running on?

df-cgdm commented 2 years ago

I'm running on a Debian 11. The problem is that /usr/bin/env python run python 2.7 rather than python 3.9 because it's the system default. The hvac library is only installed with python 3.9. When the shebang is /usr/bin/python Ansible handle it as a standard "python" module and it's using the latest installed python version. When the shebang is /usr/bin/env python it use the shebang to run the module. This behavior seems new to Ansible 6 (ansible core 2.13)

df-cgdm commented 2 years ago

The ansible documentation (https://docs.ansible.com/ansible/latest/dev_guide/testing/sanity/shebang.html) is saying

"This does not apply to Ansible modules, which should not be executable and must always use #!/usr/bin/python."

pat-s commented 2 years ago

I am seeing potentially similar issues on Alpine 3.16 (which removed Python2):

task path: /drone/src/helper-tasks/pre-tasks-onboard.yml:37
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "env: can't execute 'python': No such file or directory\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

/ansible # /usr/bin/env python
env: can't execute 'python': No such file or directory

I could fix it by 

ln -s /usr/bin/python3 /usr/bin/python
pat-s commented 2 years ago

An update on this, as my colleague and me just fell over this in another instance.

I had v4.6.6 installed, my colleague 4.7.0.

Because of the new behavior in 4.7.0, hvac python module was not installed as our pip call in the pipeline only does so for the default pip on the system (which defaults to python3.6).

To workaround this for the moment, we added another pip call to the default pip2 interpreter (if available).

Overall, I think this module should just use the default ansible python interpreter used in all other tasks rather than hardcoding it's one to /usr/bin/env python which might differ to the default one used by ansible. Also there are still many systems out there for which /usr/bin/env python points to Python2 and still using it might incur other side issues.

Thanks for maintaining this module!

pat-s commented 2 years ago

In case it helps somebody here, our current configuration to account for the installation of hvac across various OS while accounting for the python2/python3 mess:

- name: Check that /usr/bin/env python exists (for hashivault modules)
  stat:
    path: /usr/bin/python
  register: python_exists

- name: Symlink /usr/bin/env python to /usr/bin/env python3
  file:
    src: "/usr/bin/python3"
    dest: "/usr/bin/python"
    state: link
  when: not python_exists.stat.exists

- name: Check if pip2 exists
  stat:
    path: /usr/bin/pip2
  register: pip2_exists

- name: Ensure pip hvac is installed if python2 exists
  when: pip2_exists.stat.exists
  pip:
    executable: /usr/bin/pip2
    name:
      - hvac
      - hvac[parser]
    state: present

- name: Install hvac pip required for hashicorp_vault module
  become: true
  become_user: "{{ user }}"
  pip:
    name:
      - hvac
      - hvac[parser]
    state: latest
pavel-z1 commented 1 year ago

We have the same issue /usr/bin/env: ‘python’: No such file or directory Used Rocky Linux 8.7 and ansible [core 2.14.7]

MacOS 13.4.1 - the same error

TerryHowe commented 1 year ago

You'll need to make sure some Python version is in your path when you are running

pavel-z1 commented 1 year ago

Of course can be created a symlink But maybe a better solution would be replace the shebang "#!/usr/bin/env python" by "#!/usr/bin/python" as recommend ansible docs https://docs.ansible.com/ansible/latest/dev_guide/testing/sanity/shebang.html ?

ceesios commented 1 year ago

Agree,

As described here: "Begin your Ansible module with #!/usr/bin/python - this “shebang” allows ansible_python_interpreter to work. Follow the shebang immediately with # -- coding: utf-8 -- to clarify that the file is UTF-8 encoded."

"Using #!/usr/bin/env, makes env the interpreter and bypasses ansible__interpreter logic."

I am running into this issue when trying to use the modules in an pre-built k8s operator image.

This should do the trick for i in /.py; do sed -i 's/#!\/usr\/bin\/env python/#!\/usr\/bin\/python/g' $i; done

GregWhiteyBialas commented 9 months ago

I have created PR to solve the issue. Any kind of comments/help/contribution is welcomed.

TerryHowe commented 9 months ago

I think the idea was to change the shebang, not remove it.

nyetwurk commented 9 months ago

Creating and maintaining a non-pip based py-hvac/py3-hvac for alpine and a python-hvac/python3-hvac for debian would solve this for both cases.

pip combined with venv just creates more problems than either attempt to solve - imo both should be avoided in everything but a dev context.

nyetwurk commented 9 months ago

Insisting on venv requires --system-site-packages but still breaks hardcoded shebangs.

Eschewing venv (so you dont have to use env python) allows hardcoded shebangs, but requires PIP_REQUIRE_VIRTUALENV=false and possibly --break-system-packages as well.

The whole situation is a mess introduced by venv which allowed folks to use very questionable workflows - all because pip+ venv suddenly allowed developers to not bother to create native distro packaging for their python modules.

Teebor-Choka commented 4 months ago

Encountering the /usr/bin/env python: No such file or directory even though proper path and specifications exist everywhere in the venv. Any solutions, fellas? (Darwin).