search

ansible-collections / community.aws

Ansible Collection for Community AWS
GNU General Public License v3.0
188 stars 396 forks source link

unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

Closed nmoseman closed 9 months ago

nmoseman commented 1 year ago

Summary

I had good luck with the aws_ssm plugin until attempting to use it against the latest Amazon Linux AMI. Simple commands that work well with a CentOS 7 host fails when trying to run them against AMI. It appears to be something to do with interpreting shell output.

A 'raw' command like this works fine:

ansible -i inventory.aws_ec2.yaml -m 'raw' -a 'whoami' tag_role_FAKEMX

However when running the equivalent 'command' module it fails for Amazon Linux, but works on CentOS 7.

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ec2-13-58-203-89.us-east-2.compute.amazonaws.com
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
ec2-13-58-203-89.us-east-2.compute.amazonaws.com | FAILED | rc=-1 >>
failed to transfer file to /Users/username/.ansible/tmp/ansible-local-78807y4966nip/tmpqn_vbdbk /AnsiballZ_command.py:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Versus CentOS 7:

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ip-10-240-172-59.us-east-2.compute.internal
[WARNING]: Platform linux on host ip-10-240-172-59.us-east-2.compute.internal is using the discovered Python interpreter at /usr/libexec/platform-python, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.14/reference_appendices/interpreter_discovery.html for more information.
ip-10-240-172-59.us-east-2.compute.internal | CHANGED | rc=0 >>
root

From '-vvvv' output I see things like this:

<i-0cc859c89f4aaf5f4> ssm_retry: (success) (0, '\x1b[?2004h\x1b[?2004l\r\r\r\nPLATFORM\r\r\nLinux\r\r\nFOUND\r\r\n/usr/bin/python3.9\r\r\n/usr/bin/python3\r\r\nENDFOUND\r\r\n\x1b[?2004h\x1b[?2004l\r\r\r', '')
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
Using module file /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages/ansible/modules/command.py

That's a failure to find the python version. Versus: 

<i-03c9cbe64572b3eb0> ssm_retry: (success) (0, 'PLATFORM\r\r\nLinux\r\r\nFOUND\r\r\n/usr/libexec/platform-python\r\r\n/usr/bin/python2.7\r\r\n/usr/bin/python\r\r\n/usr/bin/python\r\r\nENDFOUND\r\r', '')
<ip-10-240-172-59.us-east-2.compute.internal> Python interpreter discovery fallback (pipelining support required for extended interpreter discovery)`

Where it didn't complain and seems to be successful. Notice the additional "x1b[?2004h\x1b[?2004" strings in the output.

I see the similar string in other problems, like "Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py"

Note that if I set the python interpreter it will remove the warning, but it will still error out on the "Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file Warning: or directory"

Tried this with community.aws collection versions 4.5.0, 5.2.0, and 5.3.0 and the errors is the same every time. Also tried a few different amazon.aws collection versions and had the same error.

These examples are taken from a Mac running python 3.11.2 and ansible 2.14.2, but the same errors occured in a Linux-based Execution Environment for AWX running in EKS.

Issue Type

Bug Report

Component Name

aws_ssm connection

Ansible Version

ansible [core 2.14.2]
  config file = None
  configured module search path = ['/Users/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/username/.asdf/installs/python/3.11.2/bin/ansible
  python version = 3.11.2 (main, Feb 21 2023, 11:07:56) [Clang 13.1.6 (clang-1316.0.21.2.5)] (/Users/username/.asdf/installs/python/3.11.2/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

Collection                    Version
----------------------------- -------
ansible.netcommon             4.1.0
ansible.posix                 1.5.1
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.0
awx.awx                       21.11.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.3
cisco.intersight              1.0.23
cisco.ios                     4.3.1
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.0
cisco.mso                     2.2.1
cisco.nso                     1.0.3
cisco.nxos                    4.0.1
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.4
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.10.0
community.digitalocean        1.23.0
community.dns                 2.5.0
community.docker              3.4.0
community.fortios             1.0.0
community.general             6.3.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.1.0
community.hrobot              1.7.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.7.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.6.0
community.vmware              3.3.0
community.windows             1.12.0
community.zabbix              1.9.1
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.5.0
dellemc.unity                 1.5.0
f5networks.f5_modules         1.22.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.2
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.1.2
grafana.grafana               1.1.0
hetzner.hcloud                1.9.1
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.2.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.10.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.16.2
purestorage.flashblade        1.10.0
purestorage.fusion            1.3.0
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.0
theforeman.foreman            3.8.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.7.0
vyos.vyos                     4.0.0
wti.remote                    1.0.4

# /Users/username/.ansible/collections/ansible_collections
Collection    Version
------------- -------
amazon.aws    4.3.0
community.aws 5.2.0

AWS SDK versions

❯ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.76
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.76
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

❯ ansible-config dump --only-changed
CONFIG_FILE() = None

OS / Environment

Darwin ENG-NMOSEMAN-MB 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64 i386 Darwin

and

modified version of quay.io/ansible/awx-ee:latest to include community.aws, and ssm stuff.

Steps to Reproduce

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ec2-13-58-203-89.us-east-2.compute.amazonaws.com
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
ec2-13-58-203-89.us-east-2.compute.amazonaws.com | FAILED | rc=-1 >>
failed to transfer file to /Users/username/.ansible/tmp/ansible-local-78807y4966nip/tmpqn_vbdbk /AnsiballZ_command.py:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Expected Results

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ip-10-240-172-59.us-east-2.compute.internal
ip-10-240-172-59.us-east-2.compute.internal | CHANGED | rc=0 >>
root

Actual Results


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Code of Conduct

bwells-scripps commented 1 year ago

We just ran into this issue here. It seems that Amazon Linux outputs colorized text when Ansible runs any remote shell commands which causes parsing of the result to fail. Our solution was to not use aws_ssm connection - instead we setup SSH to make connections through Session Manager: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

dennisjlee commented 1 year ago

I was facing this same issue with hosts running both Ubuntu 22.04 and Amazon Linux 2023, and I was finally able to trace the extra output to a root cause. This is due to newer versions of Bash/readline turning on the option enable-bracketed-paste by default (more details here).

I have a patch that will disable this option (will send a PR later today), which allows ansible -m ping to work on several hosts I have access to, including Ubuntu 18.04, Amazon Linux 2023, and Ubuntu 22.04.

adpavlov commented 1 year ago

https://github.com/ansible-collections/community.aws/pull/1839

Confirmed it's working with Amazon Linux

nnsense commented 1 year ago

I've ended up having the same issue switching from amazon Linux 2 to amazon Linux 2023, my error was:

service-use1-bh | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp `\"&& mkdir \"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp/ansible-tmp-1694091988.5261781-5952-41494157774678 `\" && echo ansible-tmp-1694091988.5261781-5952-41494157774678=\"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp/ansible-tmp-1694091988.5261781-5952-41494157774678 `\" ), exited with result 1, stdout output: \u001b[?2004h\u001b[?2004l\r\r\r\nmkdir: cannot create directory ‘2004h2004l’: Permission denied\r\r\n\u001b[?2004h\u001b[?2004l\r\r\r",
    "unreachable": true
}

https://github.com/ansible-collections/community.aws/pull/1839 fixes it, but it's taking ages to be merged for unknown reasons. In the meantime I've set set enable-bracketed-paste off into /etc/inputrc which is, needless to say, not a fix at all since you need to configure all servers this way, which is exactly what ansible is meant to do. In my case was just one so for now it's sorted, thanks to @dennisjlee !

Hokwang commented 9 months ago

@tremble Hi, you released v7.1.0 and its release note mentioned this issue fixed, but it is not.

tremble commented 9 months ago

@Hokwang #1839 now results in the integration tests passing, which they previously didn't. This includes tests running against the latest Amazon Linux AMIs as exposed by Amazon as the SSM Parameter /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2.

As such I can only respond with "Unable to Reproduce" at this time

It would be helpful if you could provide details of exactly what error you're seeing against which AMI. I would strongly recommend opening a new issue (comments on closed issues are very easy to overlook), with much more detail than "still doesn't work for me".