Possible usage of wrong python interpreter

[MrPink10]

It is apparently possible that ansible uses the wrong Python interpreter in an AWX job. Example:

FROM quay.io/ansible/awx-ee:latest

USER root

RUN pip install hvac

While the default Python interpreter is set here https://github.com/ansible/awx-ee/blob/devel/Containerfile#L29 In the AWX run itself, however, ansible fetches the interpreter from /usr/libexec/platform-python which points to /usr/libexec/platform-python -> ./platform-python3.6 Which then leads to the following error message

Failed to import the required Python library (hvac) on automation-job-628-b2nwk's Python /usr/libexec/platform-python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter",

It is possible to fix this via the following workaround

FROM quay.io/ansible/awx-ee:latest

USER root

RUN pip install hvac

RUN ln -sf "$(which python)" /usr/libexec/platform-python

This will force the rewrite of the link to the default Python interpreter which contain the packages installed via pip.

[shanemcd]

Hello - I think you want to use pip3.8 - we could consider making a symlink, but in general you shouldn't really be relying on an unversioned pip or python.

[MrPink10]

Hi, maybe I don't understand the answer correctly, but pip is not the problem here. With the command RUN alternatives --set python /usr/bin/python3 from your Containerfile (https://github.com/ansible/awx-ee/blob/devel/Containerfile#L29) you do not affect the Python interpreter to which point /usr/libexec/platform-python. But this is the link Ansible follow to find the Python interpreter. This has probably crept in with the change from CentOS 8 to CentOS Stream as container platform.

And yes, maybe it would be better to use the PATH to the wanted interpreter directly instead of $(which python), but this is only my little quick&dirty workaround to deal with this problem.

[relrod]

@MrPink10 Nothing should be (intentionally) using platform-python. And I don't think it is:

[rick@fedora awx-ee]$ docker run  -it --rm quay.io/ansible/awx-ee:21.11.0 ansible -m debug -a var=ansible_python_interpreter localhost
[WARNING]: No inventory was parsed, only implicit localhost is available
localhost | SUCCESS => {
    "ansible_python_interpreter": "/usr/bin/python3"
}

and we can import pip deps just fine using python which links to /usr/bin/python3:

[rick@fedora awx-ee]$ docker run  -it --rm quay.io/ansible/awx-ee:21.11.0 /bin/bash
bash-4.4$ python
Python 3.8.13 (default, Jun 24 2022, 15:27:57) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-13)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> 
bash-4.4$ ls -l /usr/bin/python*
lrwxrwxrwx. 1 root root   36 Jan 23 12:14 /usr/bin/python -> /etc/alternatives/unversioned-python
lrwxrwxrwx. 1 root root   25 Apr 28  2022 /usr/bin/python3 -> /etc/alternatives/python3
-rwxr-xr-x. 1 root root 7776 Jun 24  2022 /usr/bin/python3.8
bash-4.4$ ls -l /etc/alternatives/*python*
lrwxrwxrwx. 1 root root 16 Jan 23 12:14 /etc/alternatives/python -> /usr/bin/python3
lrwxrwxrwx. 1 root root 18 Apr 28  2022 /etc/alternatives/python3 -> /usr/bin/python3.8
lrwxrwxrwx. 1 root root 34 Apr 28  2022 /etc/alternatives/python3-man -> /usr/share/man/man1/python3.8.1.gz
lrwxrwxrwx. 1 root root 16 Jan 23 12:14 /etc/alternatives/unversioned-python -> /usr/bin/python3
lrwxrwxrwx. 1 root root 32 Jan 23 12:14 /etc/alternatives/unversioned-python-man -> /usr/share/man/man1/python3.1.gz

Additionally, we just updated the latest tag to be based on Stream 9 instead of Stream 8.

I don't think we have a bug here and I'd like to close this out unless you have additional information or still think there is something wrong.

[dakota-marshall]

Hey there! I think we are having this same issue. We have some playbooks that run entirely locally, and ansible is picking /usr/libexec/platform-python over /usr/bin/python. Granted we are using awx-ee:21.11.0, but thats due to this AWX issue.

When it gathers facts on localhost, I am getting: "discovered_interpreter_python": "/usr/libexec/platform-python"

[dakota-marshall]

Here is a cleaned up output of host facts that ran using around the same version of awx-ee that contains the references to platform-python.

This job was launched from AWX

{
  "ansible_facts": {
    "ansible_user_id": "runner",
    "ansible_user_uid": 1000,
    "ansible_user_gid": 0,
    "ansible_user_gecos": ",,,",
    "ansible_user_dir": "/home/runner",
    "ansible_user_shell": "/bin/bash",
    "ansible_real_user_id": 1000,
    "ansible_effective_user_id": 1000,
    "ansible_real_group_id": 0,
    "ansible_effective_group_id": 0,
    "ansible_system": "Linux",
    "ansible_kernel": "5.4.0-139-generic",
    "ansible_kernel_version": "#156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023",
    "ansible_machine": "x86_64",
    "ansible_python_version": "3.6.8",
    "ansible_domain": "",
    "ansible_userspace_bits": "64",
    "ansible_architecture": "x86_64",
    "ansible_userspace_architecture": "x86_64",
    "ansible_virtualization_type": "container",
    "ansible_virtualization_role": "guest",
    "ansible_virtualization_tech_guest": [
      "container",
      "VMware"
    ],
    "ansible_virtualization_tech_host": [],
    "ansible_local": {},
    "ansible_date_time": {
      "year": "2023",
      "month": "02",
      "weekday": "Friday",
      "weekday_number": "5",
      "weeknumber": "08",
      "day": "24",
      "hour": "20",
      "minute": "10",
      "second": "27",
      "epoch": "1677269427",
      "epoch_int": "1677269427",
      "date": "2023-02-24",
      "time": "20:10:27",
      "iso8601_micro": "2023-02-24T20:10:27.602507Z",
      "iso8601": "2023-02-24T20:10:27Z",
      "iso8601_basic": "20230224T201027602507",
      "iso8601_basic_short": "20230224T201027",
      "tz": "UTC",
      "tz_dst": "UTC",
      "tz_offset": "+0000"
    },
    "ansible_distribution": "CentOS",
    "ansible_distribution_release": "Stream",
    "ansible_distribution_version": "8",
    "ansible_distribution_major_version": "8",
    "ansible_distribution_file_path": "/etc/centos-release",
    "ansible_distribution_file_variety": "CentOS",
    "ansible_distribution_file_parsed": true,
    "ansible_os_family": "RedHat",
    "ansible_is_chroot": false,
    "ansible_fibre_channel_wwn": [],
    "ansible_lsb": {},
    "ansible_selinux_python_present": true,
    "ansible_selinux": {
      "status": "disabled"
    },
    "ansible_python": {
      "version": {
        "major": 3,
        "minor": 6,
        "micro": 8,
        "releaselevel": "final",
        "serial": 0
      },
      "version_info": [
        3,
        6,
        8,
        "final",
        0
      ],
      "executable": "/usr/libexec/platform-python",
      "has_sslcontext": true,
      "type": "cpython"
    },
    "ansible_system_capabilities_enforced": "True",
    "ansible_system_capabilities": [
      "i"
    ],
    "ansible_iscsi_iqn": "",
    "ansible_processor": [
      "0",
      "GenuineIntel",
      "Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz",
      "1",
      "GenuineIntel",
      "Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz",
      "2",
      "GenuineIntel",
      "Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz",
      "3",
      "GenuineIntel",
      "Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz"
    ],
    "ansible_processor_count": 4,
    "ansible_processor_cores": 1,
    "ansible_processor_threads_per_core": 1,
    "ansible_processor_vcpus": 4,
    "ansible_processor_nproc": 4,
    "ansible_memtotal_mb": 7957,
    "ansible_memfree_mb": 944,
    "ansible_swaptotal_mb": 1998,
    "ansible_swapfree_mb": 1942,
    "ansible_memory_mb": {
      "real": {
        "total": 7957,
        "used": 7013,
        "free": 944
      },
      "nocache": {
        "free": 4164,
        "used": 3793
      },
      "swap": {
        "total": 1998,
        "free": 1942,
        "used": 56,
        "cached": 5
      }
    },
    "ansible_bios_date": "11/12/2020",
    "ansible_bios_vendor": "Phoenix Technologies LTD",
    "ansible_bios_version": "6.00",
    "ansible_board_asset_tag": "NA",
    "ansible_board_name": "440BX Desktop Reference Platform",
    "ansible_board_serial": "NA",
    "ansible_board_vendor": "Intel Corporation",
    "ansible_board_version": "None",
    "ansible_chassis_asset_tag": "No Asset Tag",
    "ansible_chassis_serial": "NA",
    "ansible_chassis_vendor": "No Enclosure",
    "ansible_chassis_version": "N/A",
    "ansible_form_factor": "Other",
    "ansible_product_name": "VMware Virtual Platform",
    "ansible_product_serial": "NA",
    "ansible_product_uuid": "NA",
    "ansible_product_version": "None",
    "ansible_system_vendor": "VMware, Inc.",
    "ansible_device_links": {
      "ids": {},
      "uuids": {},
      "labels": {},
      "masters": {
        "sda3": [
          "dm-0"
        ]
      }
    },
    "ansible_uptime_seconds": 711626,
    "ansible_fips": false,
    "ansible_apparmor": {
      "status": "disabled"
    },
    "ansible_hostnqn": "",
    "ansible_pkg_mgr": "dnf",
    "ansible_service_mgr": "dumb-init",
    "gather_subset": [
      "all"
    ],
    "module_setup": true,
    "discovered_interpreter_python": "/usr/libexec/platform-python"
  },
  "warnings": [],
  "deprecations": [],
  "_ansible_verbose_override": true,
  "_ansible_no_log": false,
  "changed": false
}

[relrod]

@dakota-marshall

Are you able to reproduce this (with some minimal reproducer playbook) if you just run it in the container, outside of AWX?

If I shell into an awx-ee:21.11.0 image and run ansible-playbook on a playbook that 1) runs on localhost, and 2) spits out ansible_python_interpreter, I'm not able to reproduce this no matter how hard I try :( but I'd like to understand what's going on here.

[nitzmahone]

"Runs entirely locally" is a loaded statement :wink: - I suspect this is an inventory-related issue. Vanilla implicit localhost in core doesn't run interpreter discovery, but an explicitly-defined localhost that's missing something like ansible_python_interpreter={{ansible_playbook_python}} will run interpreter discovery, and at least for now, on RHEL8+ targets, will favor platform-python to pick up system bindings for things like selinux and dnf that aren't available on other Pythons. We've actually mostly solved the system bindings issue in recent versions of ansible-core, but interpreter discovery still currently favors the older Python on those hosts.

IIRC, some layer (AWX inventory, ansible-runner, I can't recall) was injecting an explicit localhost that didn't properly set the intepreter to match the way the core implicit one did, which is what you're likely running into.

[dakota-marshall]

Haha yea probably a poor choice of words, I was just implying we have some playbooks that only run inside the EE, so no outside factors other than AWX, ansible-core, and the EE should be a factor.

some layer (AWX inventory, ansible-runner, I can't recall) was injecting an explicit localhost that didn't properly set the intepreter to match the way the core implicit one did, which is what you're likely running into.

I agree that this is probably the issue im running into. Shelling into the container does give me /usr/lib/python. However, I havent tried running a playbook outside AWX, ill give that a shot here when I have a bit of time to mess around with it. Though I would venture to guess that this is an AWX inventory thing.