Does not support AssumeRole or MFA profiles

[phene]

AWSCurl does not support profiles that assume role (for cross-account access) or MFA.

[okigan]

@phene could you capture to this issue more details how profile with assume role is setup

[phene]

Please see the Using IAM Roles section of AWS CLI configuration guide. The role_arn and mfa_serial options are the relevant pieces of information that awscurl presently ignores.

[scraly]

We have the same needs and same issues. We use a AWS user without policies & rights, we ask to this user to assume a role. It's working corerctly with aws cli, but doen't work with awscurl. Is it possible to solve this behavior?

[tdi]

+1 to this

[okigan]

assume role would require full support for boto (right?) which seems a bit large dependency to add into awscurl

as assume role just replaces env variables to the ones corresponding to the role have you considered just running aws sts assume-role ... before running awscurl?

as shown in: https://gist.github.com/brianredbeard/035ee1419bc38a0e2d854fb828d585d7

source <(AWS_PROFILE=redbeard aws sts assume-role --role-arn arn:aws:iam::123456789123:role/OrganizationAccountAccessRole --role-session-name "DevAccount" | jq -r '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')

[tdi]

I do not think boto is big dependency - or let me rephrase it - problematic. Chance is when you use awscurl - you already have boto installed along with botocore etc. This way awscurl could simplify codebase and leveraging ofifcial's SDK config/env/profile readers and MFA and cross roles.

[okigan]

so trying to understand value proposition, if you have boto installed why use awscurl?

[phene]

The AWS CLI does not make it easy to create signed requests to Elasticsearch. That is the sole reason I use awscurl.

[legal90]

The issue was partially fixed by https://github.com/okigan/awscurl/pull/63. It works with env variables:

AWS_PROFILE=my-mfa-profile-with-assume-role awscurl $MY_HOST/test/v1/url

but doesn't work with CLI arg:

awscurl --profile my-mfa-profile-with-assume-role $MY_HOST/test/v1/url

Traceback (most recent call last):
  File "/Users/myuser/Library/Python/2.7/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/Users/myuser/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 501, in main
    inner_main(sys.argv[1:])
  File "/Users/myuser/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 471, in inner_main
    args.profile)
  File "/Users/myuser/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

[yegeniy]

Following up on https://github.com/okigan/awscurl/issues/44#issuecomment-436532408 and https://github.com/okigan/awscurl/issues/44#issuecomment-626702166:

Until #102 is merged, you can do the following to invoke awscurl with an assumed role in a profile called "temp":

1. Generate commands to configure your ~/.aws/credentials file with a "temp" profile:

aws sts assume-role --role-arn arn:aws:iam::1234567890:role/TheRoleName --role-session-name "testing" | jq -r '.Credentials | "aws configure set aws_session_token \(.SessionToken) --profile temp\naws configure set aws_access_key_id \(.AccessKeyId) --profile temp\naws configure set aws_secret_access_key \(.SecretAccessKey) --profile temp\n"'

2. Execute those three commands (just copy paste them on your terminal). Verify that there is now a profile named [temp] in your ~/.aws/credentials file

3. Finally, invoke awscurl as awscurl --profile temp <url>.

---

Assuming roles in AWS and dealing with permissions gets complicated quickly. So, don't forget about using -v flag to debug (awscurl -v).

[john-aws]

Note: the AWS_PROFILE workaround above doesn't appear to work for the external credential_process case where the profile in ~/.aws/config looks like the following (and which works correctly for all AWS SDK clients e.g. awscli):

[profile my-profile]
output = json
region = us-east-1
credential_process = cli-get-credentials name@example.com --role my-iam-role

Running:

AWS_PROFILE=my-profile awscurl $URL

Results in:

ValueError: No access key is available

[erpel]

I was going to open this as a separate issue "Using aws-adfs credentials only works with 0.27" but when researching related tickets was unsure if this is useful as a separate bug as it introduces very little new information.

TLDR: This was working with 0.27.

I started using awscurl a short while ago when 0.27 was the current version which was installed. After upgrading to 0.29 today, my workflow broke. When looking into it I realized that 0.26 also does not work. I'm pretty confident that the change from PR #146 enabled my style of credentials.

What does the credential setup look like?

Using aws-adfs (https://github.com/venth/aws-adfs), logging in to a "federated role" in AWS account A and then using entries in ~/.aws/config to assume working roles in different accounts (B).

[profile playground]
role_arn = arn:aws:iam::<ID B>:role/admin-spoke-account-role
source_profile = default
region = eu-central-1

[default]
region = eu-central-1
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::<ID A>:role/adfs-devops-role
adfs_config.adfs_host = adfs.company.com
adfs_config.session_duration = 28000
adfs_config.provider_id = urn:amazon:webservices
adfs_config.sspi = False
adfs_config.duo_factor = None
adfs_config.duo_device = None
adfs_config.adfs_user = username@company.com

aws-adfs puts temporary credentials into ~/.aws/credentials, the whole setup works fine with most tools that use boto3 and allow setting a profile.

This is the awscurl command I use to test: awscurl -v -X POST --region eu-central-1 --profile playground --service aps "$AMP_QUERY_ENDPOINT?query=up"

This whole situation seems quite complex, I found a number of related issues:

• Different workflow broken with 0.27: https://github.com/okigan/awscurl/issues/163
• Touches on a similar topic: https://github.com/okigan/awscurl/pull/102
• Reports the lack of support for an assume role workflow that's similar: https://github.com/okigan/awscurl/issues/44
• Reports --profile not working (closed as duplicate): https://github.com/okigan/awscurl/issues/89

It was quite surprising to me that setting the AWS_PROFILE variable works this differently than the --profile argument - I would not have tried this from the documentation alone. Maybe the visibility of this workaround can be improved.

[okigan]

@erpel wow - thanks for detailed report! I’ll have another look

[vazkarvishal]

Are there any plans to fix this? It is almost unusable without assumeRoles being supported.

[erpel]

It would probably be a very easy fix to introduce a new flag to "force" usage of Session(profile=profile) from botocore.session. Integrating this in a way that automatically does the right thing for all use cases with minimal manual config is probably the hard part. If the maintainers indicate that they'd be interested in a solution with an additional flag, I'd be happy to contribute an MR.

[okigan]

@erpel yes, PR, welcome -- I will have a look at integrating that.

[erpel]

Take a look at #172 please. It looks like there might be a possibility of more workflows working out of the box without a dedicated flag.

[ikarlashov]

@okigan any update on this matter? We miss this functionality too.

[okigan]

@ikarlashov some progress, botocore is becoming a standard dependency for awscurl (which was a blocker, https://github.com/okigan/awscurl/pull/205).

Now that probably does not fully solve this issue, but I am bit concerned merging PR from @erpel as I have not tested this recently and unclear if it breaks other users.

Open to ideas.

[erpel]

Maybe we can collect common and supported combinations of configs and access settings and configs. Use that to build a suite of (semi) automated tests to verify that no known setup is broken.

The hard part about this could be having access to AWS accounts with all the different setup. Might be a nightmare to set up or require collaboration of multiple users to provide input that some subset of covered cases still works.