aws_ec2 and aws sso login

danielmotaleite commented 2 years ago

Summary

I have a ansible with aws_ec2 configured and it works fine with normal aws credential ( ~/.aws/credentials and exporting the tokens). Now my company is migrating to OKTA, so i need to do aws sso login to get permissions for aws. aws cli is working fine after the login, but ansible fails, aws_ec2 report no credentials found

setting up inventory plugins
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
Loading collection amazon.aws from /home/dleite/.ansible/collections/ansible_collections/amazon/aws
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
[WARNING]:  * Failed to parse
/home/dleite/git/ansible/hosts.staging/aws_ec2.yml with
ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin: Unable to
locate credentials
  File "/usr/lib/python3/dist-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/dleite/.ansible/collections/ansible_collections/amazon/aws/plugins/inventory/aws_ec2.py", line 783, in parse
    results = self._query(regions, include_filters, exclude_filters, strict_permissions)
  File "/home/dleite/.ansible/collections/ansible_collections/amazon/aws/plugins/inventory/aws_ec2.py", line 626, in _query
    for i in self._get_instances_by_region(
  File "/home/dleite/.ansible/collections/ansible_collections/amazon/aws/plugins/inventory/aws_ec2.py", line 503, in _get_instances_by_region
    for connection, region in self._boto3_conn(regions):
  File "/home/dleite/.ansible/collections/ansible_collections/amazon/aws/plugins/inventory/aws_ec2.py", line 480, in _boto3_conn
    assumed_credentials = self._boto3_assume_role(credentials, region)
  File "/home/dleite/.ansible/collections/ansible_collections/amazon/aws/plugins/inventory/aws_ec2.py", line 438, in _boto3_assume_role
    sts_session = sts_connection.assume_role(RoleArn=iam_role_arn, RoleSessionName='ansible_aws_ec2_dynamic_inventory')
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 621, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python3/dist-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/lib/python3/dist-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/lib/python3/dist-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
[WARNING]:  * Failed to parse
/home/dleite/git/ansible/hosts.staging/aws_ec2.yml with ini

I tried exporting AWS_PROFILE, setting up in aws_ec2.yml the boto_profile and nothing work. ~/.aws/config is correct, generated by aws sso configure sso

I suspect that the code is trying to check for credentials and not allowing boto3 to run and use this new feature

Issue Type

Bug Report

Component Name

aws_ec2

Ansible Version

$  ansible --version
ansible [core 2.12.2]
  config file = /home/dleite/git/ansible/ansible.cfg
  configured module search path = ['/home/dleite/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/dleite/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
  jinja version = 2.10.1
  libyaml = True

It also failed with ansible 2.9 by the way

Collection Versions

$  ansible-galaxy collection list
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 
2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

# /usr/lib/python3/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.5.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.3  
ansible.windows               1.9.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.11.0 
check_point.mgmt              2.2.2  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.18 
cisco.ios                     2.6.0  
cisco.iosxr                   2.6.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.6.0  
cisco.mso                     1.3.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.8.2  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.2.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.2.0  
community.digitalocean        1.15.0 
community.dns                 2.0.6  
community.docker              2.1.1  
community.fortios             1.0.0  
community.general             4.4.0  
community.google              1.0.0  
community.grafana             1.3.0  
community.hashi_vault         2.2.0  
community.hrobot              1.2.2  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.3  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.6.1  
community.proxysql            1.3.1  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.17.1 
community.windows             1.9.0  
community.zabbix              1.5.1  
containers.podman             1.9.1  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.4.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.14.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.4  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.2.1  
inspur.sm                     1.3.0  
junipernetworks.junos         2.8.0  
kubernetes.core               2.2.3  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.13.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.15.1
netapp.storagegrid            21.9.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.5.1  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.6.0  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   1.6.6  
purestorage.flasharray        1.12.1 
purestorage.flashblade        1.9.0  
sensu.sensu_go                1.13.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.27.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

# /home/dleite/.ansible/collections/ansible_collections
Collection            Version
--------------------- -------
amazon.aws            3.1.1  
community.general     3.8.0  
community.hashi_vault 1.2.0

AWS SDK versions

$ $  pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.9.253
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: 
---
Name: botocore
Version: 1.16.19
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by:

Configuration

$  ansible-config dump --only-changed
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 
2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
CALLBACKS_ENABLED(/home/dleite/git/ansible/ansible.cfg) = ['timer']
DEFAULT_CALLBACK_PLUGIN_PATH(/home/dleite/git/ansible/ansible.cfg) = ['/home/dleite/git/ansible/extras/plugins/callback']
DEFAULT_FORKS(/home/dleite/git/ansible/ansible.cfg) = 10
DEFAULT_GATHERING(/home/dleite/git/ansible/ansible.cfg) = smart
DEFAULT_HASH_BEHAVIOUR(/home/dleite/git/ansible/ansible.cfg) = merge
DEFAULT_HOST_LIST(/home/dleite/git/ansible/ansible.cfg) = ['/home/dleite/git/ansible/hosts.test']
DEFAULT_LOOKUP_PLUGIN_PATH(/home/dleite/git/ansible/ansible.cfg) = ['/home/dleite/git/ansible/extras/plugins/lookup']
DEFAULT_MANAGED_STR(/home/dleite/git/ansible/ansible.cfg) = This file is managed by Ansible.%n
template: {file}
date: %Y-%m-%d %H:%M
user: {uid}
host: {host}
DEFAULT_ROLES_PATH(/home/dleite/git/ansible/ansible.cfg) = ['/home/dleite/git/ansible/roles/external', '/home/dleite/git/ansible/roles/internal']
DEFAULT_STDOUT_CALLBACK(/home/dleite/git/ansible/ansible.cfg) = debug
DISPLAY_SKIPPED_HOSTS(/home/dleite/git/ansible/ansible.cfg) = False
HOST_KEY_CHECKING(/home/dleite/git/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/home/dleite/git/ansible/ansible.cfg) = /usr/bin/python
INVENTORY_ENABLED(/home/dleite/git/ansible/ansible.cfg) = ['aws_ec2', 'ini', 'yaml']
MAX_FILE_SIZE_FOR_DIFF(/home/dleite/git/ansible/ansible.cfg) = 327680
RETRY_FILES_ENABLED(/home/dleite/git/ansible/ansible.cfg) = False

OS / Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

Steps to Reproduce

cat host.staging/aws_ec2.yml
plugin: aws_ec2

use_contrib_script_compatible_sanitization: true
boto_profile: aws_staging
regions:
  - eu-central-1
filters:
  instance-state-name : running

$ export AWS_PROFILE=aws_staging
$ ansible-inventory -i hosts.staging/ --list

Expected Results

aws sso login is working, aws s3 ls works fine, ansible aws_ec2 plugin should be able to use the same feature and list the hosts

Actual Results

[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 2.15. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
[WARNING]:  * Failed to parse /home/dleite/git/ansible/hosts.staging/aws_ec2.yml with ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin: Unable to locate credentials
[WARNING]:  * Failed to parse /home/dleite/git/ansible/hosts.staging/aws_ec2.yml with ini plugin: Invalid host pattern '---' supplied, '---' is normally a sign this is a YAML file.
[WARNING]:  * Failed to parse /home/dleite/git/ansible/hosts.staging/aws_ec2.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]: Unable to parse /home/dleite/git/ansible/hosts.staging/aws_ec2.yml as an inventory source
(...other inventory list---)

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description:

[plugins/inventory/aws_ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/aws_ec2.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

danielmotaleite commented 2 years ago

i found in the net this reference for the same problem: https://www.reddit.com/r/ansible/comments/qql86z/how_can_i_use_aws_sso_with_ansible_aws_ec2/ Probably the mujahidk is not really using aws sso login and so just exporting the profile with normal tokens

alinabuzachis commented 2 years ago

@danielmotaleite Thank you for raising this. I don't have much experience with aws_ec2, but it seems to me that it does not support SSO. I'll wait for @jillr's confirmation to be sure and further clarification.

danielmotaleite commented 2 years ago

If not supported, please upgrade this issue to a feature request, we use OKTA for aws authentication, so this is required... right now, as workaround, we have to login to okta, choose the aws, profile, re-authenticate and copy the AWS_* tokens ... it works, but lot more work and annoying

tremble commented 2 years ago

@danielmotaleite

The problem is that SFAIK none of the maintainers are using AWS's SSO offering, so we can't even be sure why it's not working, or test any fixes. I, for example, have direct SAML integration[1] in place, instead of using the additional AWS SSO layer. I would have expected that authentication using the profile should have worked, but since I've not used AWS SSO I don't know what differences there might be between the standard integration and AWS SSO.

What I do find odd is where the failure is occurring, it seems to imply that the iam_role_arn parameter has been set, which doesn't match with the reproducer example you've provided.

If anyone with AWS SSO experience/access is able to produce a patch, we'll happily review the patch and can try and get it merged if it doesn't break existing functionality.

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml.html which predates the full SSO offering by a couple of years.

gravesm commented 2 years ago

@danielmotaleite From what I can tell, boto3 support for AWS SSO was not added until 1.14.0. What happens if you upgrade your boto3 version?

danielmotaleite commented 2 years ago

@gravesm that is the problem, i upgraded the boto3 to the latest version and aws sso login works with aws_ec2 without issues

I didn't noticed that i was below the boto3 recommended version. Maybe aws_ec2 should warn about a unsupported boto3 version, that would trigger a fix by the user ?

You can reuse this to add this minimum version check, if not, i think this can be closed. Thanks for the help!

alinabuzachis commented 2 years ago

@tremble Is it worth adding a note for sso concerning boto3 version? The only branch requiring boto3 >= 1.9 is stable-1.5 at the moment.

tremble commented 2 years ago

@alinabuzachis It's documented in https://github.com/ansible-collections/amazon.aws#aws-sdk-version-compatibility since the 2.0 release (where we required boto3 >= 1.15.0)

That said, it looks like I didn't update the inventory plugin docs to require a specific version of botocore/boto3. I think we should update this, and I wouldn't consider this a 'breaking' change since it's only documentation and we already say that the collection as a whole requires the botocore/boto3 versions.

I also think @danielmotaleite's suggestion that the remaining (non-module) plugins should also spit out warnings is a good idea, in combination with updating the inventory plugin docs.

tremble commented 2 years ago

As a bare minimum solution #819

I agree that we should also emit a warning where the minimum requirements aren't met so I'm going to leave this open for now.

jjshoe commented 2 years ago

This can actually happen to this day, if you're authed in a gov cloud region, and try to get a list from a commercial region. It seems like a warning should be thrown/regions disabled, when trying to get a list of contents for a region you don't have access to?

Grab the region they authed with, if gov cloud, disable all commercial. I'm guessing govcloud is disabled by default, and it's not an issue in the other direction.

my_session = boto3.session.Session()
my_region = my_session.region_name

ansible-collections / amazon.aws