USER_SITE path not included with lookups if the directory didn't exist when starting the playbook

SUMMARY

If you install a python module, like with ansible.builtin.pip during a play and it is installed to the user directory and that directory didn't exist prior to the playbook being started, the USER_SITE path is not included in the paths that python searches when running lookups. But if you run a module, the USER_SITE path is included.

From my troubleshooting, I am currently unable to confirm if this is an issue with community.hashi_vault, an issue with ansible, or an issue with the awx-ee container.

ISSUE TYPE

Bug Report

COMPONENT NAME

All lookups that require a python module that isn't already installed.

ANSIBLE VERSION

ansible [core 2.15.5]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.17 (main, Aug  9 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /runner/.ansible/collections/ansible_collections
Collection            Version
--------------------- -------
community.hashi_vault 6.0.0

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

quay.io/ansible/awx-ee:23.3.1

STEPS TO REPRODUCE

Have a lookup that requires a python module to be installed. Ensure that the user site directory where user instances of python modules would be installed doesn't exist at the time that the playbook is started. Install the required python module to the user space during the playbook and then attempt to use a lookup plugin that relies on that module.

---
- hosts: all
  connection: local
  gather_facts: false
  tasks:
    - name: Install prereq python modules
      ansible.builtin.pip:
        name: hvac
        extra_args: --user

    - name: Collect secret
      community.hashi_vault.vault_kv1_get:
        path: "{{ ansible_hashi_vault_path }}"
        url: "{{ ansible_hashi_vault_addr }}"
        token: "{{ ansible_hashi_vault_token }}"
        engine_mount_point: "{{ ansible_hashi_vault_engine_mount_point }}"
      register: vault_secrets

    - name: Output module secrets:
      ansible.builtin.debug:
        msg: "{{ vault_secrets }}"

    - name: Output debug info
      ansible.builtin.debug:
        msg: "{{ lookup('community.hashi_vault.vault_kv1_get', ansible_hashi_vault_path) }}"

EXPECTED RESULTS

The expected results are that both the module and lookup that are attempting to pull a secret from the vault return the same information.

ACTUAL RESULTS

The module is able to successfully pull the secret, but the lookup fails stating that the "hvac" module is not installed. From troubleshooting, this appears to be due to the path list that python is using not being updated mid-playbook for lookups, but they are for modules.

{"msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.vault_kv1_get'. Error was a <class 'ansible.errors.AnsibleError'>, original message: This plugin requires the 'hvac' Python library. This plugin requires the 'hvac' Python library"}

Hi @tylergmuir , thanks for this detailed report.

From my troubleshooting, I am currently unable to confirm if this is an issue with community.hashi_vault, an issue with ansible, or an issue with the awx-ee container.

This is an issue with ansible-core, since collections do not control the python loading mechanism. However, I think it's also an expected behavior due to Python itself.

It may help to first understand the difference between the Ansible controller (where ansible runs) vs. the target host (the host that a module runs against). Even if the target host is the controller, the difference is important; specifically it's important because modules run outside the controller process even if they are run on the same machine.

Because it's a new process, python can reload the search paths and find the module you just installed.

Lookups on the other hand, like all other plugins, run inside the controller process, so the search paths have already been loaded and won't be updated unless some code in the process specifically does so.

In short: this behavior is expected and unlikely to be considered a bug.

Since you're using Execution Environments, the way to address this is to build your EE with hvac installed as part of the build process. I don't really use EE's right now, so it's a bit beyond what I can help with directly.

I recommend posting on the new Ansible Community Forum where you should be able to get a more detailed answer about the behavior above, as well as help with building and using Execution environments.

ansible-collections / community.hashi_vault