aws_ec2 inventory does not work with MFA OTP

Summary

aws_ec2 fails to generate an inventory when using AWS credentials that require MFA.

❯ cat ~/.aws/config
[profile default]
region=us-east-1

[profile test]
role_arn=arn:aws:iam::99999999:role/switch-user
mfa_serial=arn:aws:iam::88888888:mfa/u
source_profile=default

❯ env|grep AWS
AWS_DEFAULT_PROFILE=test
AWS_PROFILE=test
AWS_EB_PROFILE=test

❯ aws sts get-caller-identity|cat
{
    "UserId": "XXXXXXXX:botocore-session-123456",
    "Account": "88888888",
    "Arn": "arn:aws:sts::99999999:assumed-role/switch-user/botocore-session-123456"
}

Issue Type

Bug Report

Component Name

ansible_collections.amazon.aws.plugins.inventory.aws_ec2

Ansible Version

❯ ansible --version
ansible 2.10.17
  config file = /Users/u/.ansible/ansible.cfg
  configured module search path = ['/Users/u/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages/ansible
  executable location = /Users/u/python-environments/ansible-3.4.0/bin/ansible
  python version = 3.10.6 (main, Aug 30 2022, 05:12:36) [Clang 13.1.6 (clang-1316.0.21.2.5)]

Collection Versions

❯ ansible-galaxy collection list

# /Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.0
ansible.netcommon             1.5.0
ansible.posix                 1.2.0
ansible.utils                 2.1.0
ansible.windows               1.5.0
arista.eos                    1.3.0
awx.awx                       17.1.0
azure.azcollection            1.5.0
check_point.mgmt              2.0.0
chocolatey.chocolatey         1.1.0
cisco.aci                     2.0.0
cisco.asa                     1.0.4
cisco.intersight              1.0.15
cisco.ios                     1.3.0
cisco.iosxr                   1.2.1
cisco.meraki                  2.2.1
cisco.mso                     1.1.0
cisco.nso                     1.0.3
cisco.nxos                    1.4.0
cisco.ucs                     1.6.0
cloudscale_ch.cloud           2.1.0
community.aws                 1.5.0
community.azure               1.0.0
community.crypto              1.6.2
community.digitalocean        1.3.0
community.docker              1.6.0
community.fortios             1.0.0
community.general             2.5.2
community.google              1.0.0
community.grafana             1.2.1
community.hashi_vault         1.1.3
community.hrobot              1.1.1
community.kubernetes          1.2.1
community.kubevirt            1.0.0
community.libvirt             1.0.1
community.mongodb             1.2.1
community.mysql               1.4.1
community.network             2.2.0
community.okd                 1.1.2
community.postgresql          1.3.0
community.proxysql            1.0.0
community.rabbitmq            1.0.3
community.routeros            1.1.0
community.skydive             1.0.0
community.sops                1.0.6
community.vmware              1.10.0
community.windows             1.3.0
community.zabbix              1.3.0
containers.podman             1.5.0
cyberark.conjur               1.1.0
cyberark.pas                  1.0.6
dellemc.openmanage            3.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.9.1
fortinet.fortimanager         2.0.2
fortinet.fortios              1.1.9
frr.frr                       1.0.3
gluster.gluster               1.0.1
google.cloud                  1.0.2
hetzner.hcloud                1.4.3
ibm.qradar                    1.0.3
infinidat.infinibox           1.2.4
inspur.sm                     1.1.4
junipernetworks.junos         1.3.0
kubernetes.core               1.2.1
mellanox.onyx                 1.0.0
netapp.aws                    20.9.0
netapp.elementsw              20.11.0
netapp.ontap                  21.6.0
netapp_eseries.santricity     1.2.8
netbox.netbox                 2.1.0
ngine_io.cloudstack           2.1.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.4.0
openvswitch.openvswitch       1.2.0
ovirt.ovirt                   1.4.2
purestorage.flasharray        1.8.0
purestorage.flashblade        1.6.0
sensu.sensu_go                1.10.0
servicenow.servicenow         1.0.4
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.17.0
theforeman.foreman            1.5.1
vyos.vyos                     1.1.1
wti.remote                    1.0.1

AWS SDK versions

❯ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.24.68
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.27.68
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

❯ ansible-config dump --only-changed|cat
ANSIBLE_FORCE_COLOR(/Users/u/.ansible/ansible.cfg) = True
ANSIBLE_NOCOWS(/Users/u/.ansible/ansible.cfg) = False
ANSIBLE_SSH_ARGS(/Users/u/.ansible/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=900s -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey,keyboard-interactive
ANSIBLE_SSH_CONTROL_PATH(/Users/u/.ansible/ansible.cfg) = ~/.ssh/ansible-%%r@%%h:%%p
CACHE_PLUGIN(/Users/u/.ansible/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/Users/u/.ansible/ansible.cfg) = ~/.ansible/cache/fact
CACHE_PLUGIN_TIMEOUT(/Users/u/.ansible/ansible.cfg) = 86400
COLOR_CHANGED(/Users/u/.ansible/ansible.cfg) = yellow
COLOR_DEBUG(/Users/u/.ansible/ansible.cfg) = dark gray
COLOR_DEPRECATE(/Users/u/.ansible/ansible.cfg) = purple
COLOR_DIFF_ADD(/Users/u/.ansible/ansible.cfg) = green
COLOR_DIFF_LINES(/Users/u/.ansible/ansible.cfg) = cyan
COLOR_DIFF_REMOVE(/Users/u/.ansible/ansible.cfg) = red
COLOR_ERROR(/Users/u/.ansible/ansible.cfg) = red
COLOR_HIGHLIGHT(/Users/u/.ansible/ansible.cfg) = white
COLOR_OK(/Users/u/.ansible/ansible.cfg) = green
COLOR_SKIP(/Users/u/.ansible/ansible.cfg) = cyan
COLOR_UNREACHABLE(/Users/u/.ansible/ansible.cfg) = red
COLOR_VERBOSE(/Users/u/.ansible/ansible.cfg) = blue
COLOR_WARN(/Users/u/.ansible/ansible.cfg) = bright purple
DEFAULT_FORKS(/Users/u/.ansible/ansible.cfg) = 20
DEFAULT_GATHERING(/Users/u/.ansible/ansible.cfg) = smart
DEFAULT_HOST_LIST(/Users/u/.ansible/ansible.cfg) = ['/Users/u/.ansible/inventory']
DEFAULT_INTERNAL_POLL_INTERVAL(/Users/u/.ansible/ansible.cfg) = 0.001
DEFAULT_INVENTORY_PLUGIN_PATH(/Users/u/.ansible/ansible.cfg) = ['/Users/u/python-environments/ansible-3.4.0/lib/python3.10/site-packages/ansible_collections']
DEFAULT_POLL_INTERVAL(/Users/u/.ansible/ansible.cfg) = 10
DEFAULT_REMOTE_USER(/Users/u/.ansible/ansible.cfg) = u
DEFAULT_SCP_IF_SSH(/Users/u/.ansible/ansible.cfg) = smart
DEFAULT_SSH_TRANSFER_METHOD(/Users/u/.ansible/ansible.cfg) = smart
DEFAULT_STDOUT_CALLBACK(env: ANSIBLE_STDOUT_CALLBACK) = yaml
DEFAULT_TRANSPORT(/Users/u/.ansible/ansible.cfg) = ssh
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /Users/u/.vault_pass.txt
DISPLAY_ARGS_TO_STDOUT(/Users/u/.ansible/ansible.cfg) = False
HOST_KEY_CHECKING(/Users/u/.ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/Users/u/.ansible/ansible.cfg) = auto
INVENTORY_CACHE_ENABLED(/Users/u/.ansible/ansible.cfg) = True
INVENTORY_CACHE_PLUGIN(/Users/u/.ansible/ansible.cfg) = jsonfile
INVENTORY_ENABLED(/Users/u/.ansible/ansible.cfg) = ['amazon.aws.aws_ec2', 'aws_ec2', 'host_list', 'script', 'yaml', 'ini', 'auto']
PARAMIKO_LOOK_FOR_KEYS(/Users/u/.ansible/ansible.cfg) = False
PERSISTENT_CONNECT_TIMEOUT(/Users/u/.ansible/ansible.cfg) = 30
RETRY_FILES_ENABLED(env: ANSIBLE_RETRY_FILES_ENABLED) = True
RETRY_FILES_SAVE_PATH(env: ANSIBLE_RETRY_FILES_SAVE_PATH) = /Users/u/.ansible/retry
USE_PERSISTENT_CONNECTIONS(env: ANSIBLE_USE_PERSISTENT_CONNECTIONS) = True

OS / Environment

❯ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 12.5.1 (21G83)
      Kernel Version: Darwin 21.6.0
      Boot Mode: Normal
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled

Steps to Reproduce

❯ cat ~/.ansible/inventory/testing.aws_ec2.yaml

plugin: amazon.aws.aws_ec2
boto_profile: test
hostnames:
  - tag:Name
regions:
  - "us-east-1"
keyed_groups:
  - key: vpc_id
  - key: tags.Service
    parent_group: "{{ vpc_id }}"
    prefix: test

❯ ansible-inventory -i ~/.ansible/inventory/testing.aws_ec2.yaml --graph
Enter MFA code for arn:aws:iam::88888888:mfa/u:
Enter MFA code for arn:aws:iam::88888888:mfa/u:
Enter MFA code for arn:aws:iam::88888888:mfa/u:
Enter MFA code for arn:aws:iam::88888888:mfa/u:

Expected Results

I expect that when I enter my OTP at the Enter MFA code prompt the plugin should continue and generate an inventory. Instead it repeatedly re-prompts for the OTP until giving up. I am 100% confident I am entering the correct code. All other tooling I use with boto3 works with my .aws/config.

Actual Results

[WARNING]:  * Failed to parse /Users/u/.ansible/inventory/testing.aws_ec2.yaml with
ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin: Failed to describe instances: An error occurred
(AccessDenied) when calling the AssumeRole operation: MultiFactorAuthentication failed with invalid MFA one time
pass code.

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansible-collections / amazon.aws

aws_ec2 inventory does not work with MFA OTP #1091