Ansible fails with "worker found in a dead state" when we try to access amazon.aws.aws_secret on OSX

cgeorgelee commented 2 years ago

Summary

When I try to lookup amazon.aws.aws_secret on OSX it fails with

"ERROR! A worker was found in a dead state"

The script run on AWS Linux 2 but does not run locally. I have made it grab the secrets before but now the script fails consistently. I have also run with

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Have also tried with the flags -vvvv -f1 but the debug does not print additional useful information and the -f1 flag does not help either.

Issue Type

Bug Report

Component Name

amazon.aws.aws_secret

Ansible Version

$ ansible --version
ansible [core 2.11.11]

Collection Versions

$ ansible-galaxy collection list

# /Users/my.user/.ansible/collections/ansible_collections
Collection    Version
------------- -------
amazon.aws    3.2.0
community.aws 3.2.1

# /Users/my.user/.pyenv/versions/3.8.12/envs/mycorpopsenv/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.1
ansible.netcommon             2.5.0
ansible.posix                 1.3.0
ansible.utils                 2.4.3
ansible.windows               1.8.0
arista.eos                    2.2.0
awx.awx                       19.4.0
azure.azcollection            1.10.0
check_point.mgmt              2.2.0
chocolatey.chocolatey         1.1.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.6.0
cisco.iosxr                   2.6.0
cisco.meraki                  2.5.0
cisco.mso                     1.2.0
cisco.nso                     1.0.3
cisco.nxos                    2.8.2
cisco.ucs                     1.6.0
cloudscale_ch.cloud           2.2.0
community.aws                 1.5.0
community.azure               1.1.0
community.crypto              1.9.8
community.digitalocean        1.13.0
community.docker              1.10.2
community.fortios             1.0.0
community.general             3.8.3
community.google              1.0.0
community.grafana             1.3.0
community.hashi_vault         1.5.0
community.hrobot              1.2.1
community.kubernetes          1.2.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.2
community.network             3.0.0
community.okd                 1.1.2
community.postgresql          1.6.0
community.proxysql            1.3.0
community.rabbitmq            1.1.0
community.routeros            1.2.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.0
community.windows             1.8.0
community.zabbix              1.5.1
containers.podman             1.9.0
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            3.6.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.13.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.3
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.0
inspur.sm                     1.3.0
junipernetworks.junos         2.8.0
kubernetes.core               1.2.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.1
netapp.elementsw              21.7.0
netapp.ontap                  21.14.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.4.0
ngine_io.cloudstack           2.2.2
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.5.3
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.11.0
purestorage.flashblade        1.8.1
sensu.sensu_go                1.12.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.26.0
theforeman.foreman            2.2.0
vyos.vyos                     2.6.0
wti.remote                    1.0.3

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.22.0
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/my.user/.pyenv/versions/3.8.12/envs/mycorpopsenv/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.25.0
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/my.user/.pyenv/versions/3.8.12/envs/mycorpopsenv/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CALLBACKS_ENABLED(/Users/my.user/mycorp/views/myproj-ops/ansible.cfg) = ['profile_tasks', 'timer', 'slack']
DEFAULT_REMOTE_USER(/Users/my.user/mycorp/views/myproj-ops/ansible.cfg) = ec2-user
RETRY_FILES_ENABLED(/Users/my.user/mycorp/views/myproj-ops/ansible.cfg) = False

OS / Environment

macOS Monterey: 12.4 (21F79)

Steps to Reproduce

---
- name: test pulling from AWS Secrets Manager
  hosts: localhost
  connection: local
  run_once: true
  vars:
    ansible_python_interpreter: "{{ ansible_playbook_python }}"

  tasks:
  - name: Pull the passwords into a variable called "vault" from the AWS Secrets Manager
    set_fact:
      vault: "{{ lookup('amazon.aws.aws_secret', 'ops/production', on_denied='warn') | from_json }}"

Expected Results

Should return json results from AWS Secrets Manager.

Actual Results

$ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES; ansible-playbook test_vault_pull.yml -vvvv -f1
ansible-playbook [core 2.11.11]
  config file = /Users/my.user/mycorp/views/myproj-ops/ansible.cfg
  configured module search path = ['/Users/my.user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/my.user/.pyenv/versions/3.8.12/envs/mycorpopsenv/lib/python3.8/site-packages/ansible
  ansible collection location = /Users/my.user/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/my.user/.pyenv/versions/mycorpopsenv/bin/ansible-playbook
  python version = 3.8.12 (default, Feb 16 2022, 15:50:20) [Clang 12.0.0 (clang-1200.0.32.29)]
  jinja version = 3.1.1
  libyaml = True
Using /Users/my.user/mycorp/views/myproj-ops/ansible.cfg as config file
setting up inventory plugins

... cut_a_bunch_of_lines ...

ok: [localhost]
META: ran handlers

TASK [Pull the passwords into a variable called "vault" from the AWS Secrets Manager] **************************************
task path: /Users/my.user/mycorp/views/myproj-ops/test_vault_pull.yml:10
Thursday 30 June 2022  00:29:54 -0700 (0:00:00.798)       0:00:01.523 *********
Loading collection amazon.aws from /Users/my.user/.ansible/collections/ansible_collections/amazon/aws
ERROR! A worker was found in a dead state

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description: None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

Akasurde commented 2 years ago

@cgeorgelee Thanks for reporting this issue. This issue is already known and documented here. There is very little Ansible can do regarding this as this issue is related to the macOS forking model. You can read more about this in this thread. I hope this helps.

Thanks,

ansible-collections / amazon.aws