Use botocore to support SSO profiles

[iainelder]

With this change, awscurl uses botocore to fetch the temporary credentials for SSO profiles from the AWS CLI v2 credential cache.

awscurl used to create a session with the get_session function. The get_session function is poorly documented, but according to the botocore README it appears to load the default profile.

To load a named profile we need to use the Session object constructor and set the profile keyword argument. To load the default profile in this way, the profile keyword is set to None.

The load_aws_config function is tested in load_aws_config_test.py. It has no tests around the botocore behavior, and I don't see how to add them easily.

As botocore already has support for SSO profiles, at the application level this is a trivial change. Botocore has its own tests for handling credentials.

Manual testing

In my AWS config file (~/.aws/config) I have an AWS SSO profile like this:

[profile example]
sso_start_url = https://d-1111111111.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 111111111111
sso_role_name = AdministratorAccess

The AWS shared credentials file (~/.aws/credentials) has no corresponding credentials for the AWS SSO profile. Instead it has "default" credentials used to simulate the existing support for EC2 instance credentials.

[default]
aws_access_key_id = EXAMPLE
aws_secret_access_key = EXAMPLE

awscurl is installed with the botocore optional dependency.

pipx uninstall awscurl && pipx install ".[awslibs]"

I use this awscurl command to look up an S3 bucket using the AWS SSO profile. The actual AWS API operation doesn't matter. The important part is that the profile selected uses AWS SSO.

awscurl \
--verbose \
--profile example \
--region=eu-west-1 \
--service=s3 \
http://example.s3.eu-west-1.amazonaws.com/

Before the change

awscurl fails with an AttributeError because it fails to find an access key. The output starting from loading the botocore package looks like this:

'loading botocore package'
Traceback (most recent call last):
  File "/home/isme/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

After the change

awscurl successfully sends a request and receives a response from AWS. The abridged output starting from the loading the botocore package looks like this:

'loading botocore package'
''
('\n'
 'CANONICAL REQUEST = GET\n'
 '/\n'
 '\n'
 'host:example.s3.eu-west-1.amazonaws.com\n'
 'x-amz-date:20210610T151322Z\n'
 'x-amz-security-token:EXAMPLE\n'
 '\n'
 'host;x-amz-date;x-amz-security-token\n'
 'EXAMPLE')[
[...]]
'Response code: 301\n'
{'x-amz-bucket-region': 'us-east-1', 'x-amz-request-id': 'EXAMPLE', 'x-amz-id-2': 'EXAMPLE', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 10 Jun 2021 15:13:21 GMT', 'Server': 'AmazonS3'}

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>s3.amazonaws.com</Endpoint><Bucket>example</Bucket><RequestId>EXAMPLE</RequestId><HostId>EXAMPLE</HostId></Error>]

I run one more command that uses the default credentials to check that the existing support for instance credentials still works. (I haven't had time to test this on a real EC2 instance.)

awscurl \
--verbose \
--region=eu-west-1 \
--service=s3 \
http://example.s3.eu-west-1.amazonaws.com/

The request and response are generated in the same way. (Surprisingly it doesn't seem to matter to AWS at this point whether the credentials are valid or not!)

'Response code: 301\n'
{'x-amz-bucket-region': 'us-east-1', 'x-amz-request-id': 'EXAMPLE', 'x-amz-id-2': 'EXAMPLE', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 10 Jun 2021 16:20:54 GMT', 'Server': 'AmazonS3'}

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>s3.amazonaws.com</Endpoint><Bucket>example</Bucket><RequestId>EXAMPLE</RequestId><HostId>EXAMPLE</HostId></Error>

[okigan]

very cool -- any thoughts how to add a unit test for it?

[iainelder]

I see there are some tests around the custom profile handling in https://github.com/okigan/awscurl/blob/master/tests/load_aws_config_test.py.

How do I run the existing unit tests? I don't see anything in the README about it.

[okigan]

What a great point about having a section on this in readme - I’ll update.

For now check out the GitHub actions file that shows commands to run:

https://github.com/okigan/awscurl/blob/f60961b4c6725a21bc4532a04d36c7f756636054/.github/workflows/pythonapp.yml#L25

On Jun 11, 2021, at 3:47 AM, Iain Samuel McLean Elder @.***> wrote:

 I see there are some tests around the custom profile handling in https://github.com/okigan/awscurl/blob/master/tests/load_aws_config_test.py.

How do I run the existing unit tests? I don't see anything in the README about it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[iainelder]

Hi, @okigan , sorry, I didn't get around to running the unit tests.

I see you have already merged this to master.

What do we need to do to get this feature into the version available from PyPI?

[okigan]

@iainelder unit tests still would be great for this - it's a bit unique feature and unit test would make sure it does not get broken by other pull requests.

I am working through another PR - so PyPI would get updated after that (so you still have time)

[okigan]

@iainelder FYI, PyPi (https://pypi.org/project/awscurl/) is updated -- test PR is still welcome to keep the feature working

[iainelder]

Hi @okigan, thanks for the update. It's not immediately clear to me how to test it but I'll have a think about it and let you know.