"ERROR! A worker was found in a dead state" When using aws_secret lookup module on MacOS

fishfacemcgee commented 1 year ago

Summary

This issue, as best I can tell is similar if not identical to #907 and #1171. However, in #907, as I best I can tell, the repo collaborator who was investigating the issue missed that the reporter had the MacOS fork flag enabled and was still running into the issue. In #1171, the issue reporter was able to resolve the issue by recreating their virtual environment. That has not worked for me.

Issue Type

Bug Report

Component Name

lookup/aws_secret.py

Ansible Version

$ ansible --version
ansible [core 2.14.1]
  config file = /opt/formstack/server-playbooks/ansible.cfg
  configured module search path = ['/Users/brian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/formstack/server-playbooks/env/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/brian/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/formstack/server-playbooks/env/bin/ansible
  python version = 3.9.13 (main, Sep 15 2022, 15:08:15) [Clang 12.0.0 (clang-1200.0.31.1)] (/opt/formstack/server-playbooks/env/bin/python)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    5.1.0
ansible.netcommon             4.1.0
ansible.posix                 1.4.0
ansible.utils                 2.8.0
ansible.windows               1.12.0
arista.eos                    6.0.0
awx.awx                       21.10.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.3.1
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.1
cisco.intersight              1.0.22
cisco.ios                     4.0.0
cisco.iosxr                   4.0.3
cisco.ise                     2.5.9
cisco.meraki                  2.13.0
cisco.mso                     2.1.0
cisco.nso                     1.0.3
cisco.nxos                    4.0.1
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.3
community.aws                 5.0.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.9.0
community.digitalocean        1.22.0
community.dns                 2.4.2
community.docker              3.3.1
community.fortios             1.0.0
community.general             6.1.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.0.0
community.hrobot              1.6.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.1
community.proxysql            1.4.0
community.rabbitmq            1.2.3
community.routeros            2.5.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.5.0
community.vmware              3.2.0
community.windows             1.11.1
community.zabbix              1.9.0
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.14
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.21.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.1
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.0.2
grafana.grafana               1.1.0
hetzner.hcloud                1.9.0
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.10.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.2.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.21.0
netapp.elementsw              21.7.0
netapp.ontap                  22.0.1
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.3.1
netbox.netbox                 3.9.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.2
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.15.0
purestorage.flashblade        1.10.0
purestorage.fusion            1.2.0
sensu.sensu_go                1.13.1
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.31.4
theforeman.foreman            3.7.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.3.1
vyos.vyos                     4.0.0
wti.remote                    1.0.4

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /opt/formstack/server-playbooks/env/lib/python3.9/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.26.52
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /opt/formstack/server-playbooks/env/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.52
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /opt/formstack/server-playbooks/env/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = /opt/formstack/server-playbooks/ansible.cfg
DEFAULT_BECOME_FLAGS(/opt/formstack/server-playbooks/ansible.cfg) = -i
DEFAULT_GATHERING(/opt/formstack/server-playbooks/ansible.cfg) = smart
DEFAULT_MANAGED_STR(/opt/formstack/server-playbooks/ansible.cfg) = Ansible managed template.  Do not edit directly.
DEFAULT_TIMEOUT(/opt/formstack/server-playbooks/ansible.cfg) = 5
DEFAULT_VAULT_PASSWORD_FILE(/opt/formstack/server-playbooks/ansible.cfg) = /opt/formstack/server-playbooks/vault-password.txt
HOST_KEY_CHECKING(/opt/formstack/server-playbooks/ansible.cfg) = False

OS / Environment

Control Node

MacOS 12.6.2 Python 3.9.13 (running via venv)

Managed Node

Amazon Linux 2

Steps to Reproduce

---
- name: fetch secretsmanager secret in the current region
  set_fact:
    raw_secrets: "{{ lookup('amazon.aws.aws_secret', '/path/to/config', aws_profile='testing_env', region='us-east-1', bypath=true) }}"
  tags:
    - config

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ansible-playbook -i inventories/testing.aws_ec2.yml playbook.yml --tags config

Expected Results

I expected the task to succeed, with the fact set with the results of the lookup.

Actual Results

TASK [config : fetch secretsmanager secret in the current region] **********************************************************************************************************************************************************************************
ERROR! A worker was found in a dead state

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibullbot commented 1 year ago

Files identified in the description: None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

fishfacemcgee commented 1 year ago

After doing some additional triage, I've confirmed that the OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES environment variable has zero impact, but, for reasons I have yet to understand, no_proxy="*" does. To clarify/reiterate, this command does not work:

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ansible-playbook -i inventories/testing.aws_ec2.yml playbook.yml --tags config

but this command does:

no_proxy="*" ansible-playbook -i inventories/testing.aws_ec2.yml playbook.yml --tags config

alinabuzachis commented 1 year ago

@fishfacemcgee Thank you for reporting this. I guess there's a known issue with Python app forking on macOS. Please have a look here https://github.com/python/cpython/issues/74570#issuecomment-1093748531 for a detailed explanation. Since the error is not originating from the collection, I'm closing the issue, but feel free to reopen if something is unclear.

fishfacemcgee commented 1 year ago

@alinabuzachis I understand that this is an issue with CPython and MacOS, but are you also saying that the Ansible AWS Collection does not/has not/will not implement mitigations for lower level platform issues in its modules or even document the compatibility issue?

Going by the comment you linked, it doesn't appear that the CPython developers are willing/able to do anything at its level to fix this either, instead recommending to use the no_proxy environment variable. If it's a policy for this Collection to not implement any sort of platform-targeting compatibility fixes, it seems like it'd at least be worthwhile to have the issue documented.

In any case, thank you for the clarification/information.

ansible-collections / amazon.aws