refreshing sts role assumption credentials for long running operations

[kapilt]

boto3 looks like it wires in credential refreshing when its using instance roles by default. The issue is how how to setup the same behavior when doing sts role assumption, to refresh credentials (i'm initially starting with an instance role). Atm the only way to pass the sts assumed role credentials is in via explicitly creating a session/client with key/secret/token, but that results in the creation of a default Credentials class without the refresh behavior. There doesn't seem to be a clean way to provide a RefreshableCredential acquired with sts token assumption to a client.

[kapilt]

by clean way, i mean not poking at botocore session internals for example, this is what i'm currently doing to make this work https://gist.github.com/kapilt/ac8e222081f63ba64e93

[russellballestrini]

I'm not a core dev but this is how I'm dealing with AWS access/secret keys, regions, etc: https://github.com/russellballestrini/botoform/blob/master/botoform/util.py#L18-L87

Edit: This class makes it possible to switch between the AWS config profile/region programatically.

[mtdowling]

There is an AssumeRoleProvider for credentials in Botocore that might help here: https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L589. I'll try to get more information on this and see if I can provide an example.

Please also feel free to chime in to the related discussion happening on Botocore about how we can improve assume role support: https://github.com/boto/botocore/issues/761

[ghost]

+1

[ghost]

@russellballestrini do you have a gist of consuming that code? I want to use it for s3 client for downloading and uploading files, a process that goes on for hours if not days.

EDIT: I don't see how the refresh is being auto refreshed.

[russellballestrini]

@BardiaAfshin the code that I linked was never tested with "assuming a role". That class just makes it possible to switch between the AWS config profile/region programatically.

Sorry about that.

[ghost]

I see. I'm surprised to not see the auto refresh solved already.

[babartareen]

+1

[JoeyG1973]

Anybody have any clue how to get cert refresh with assume_role_with_saml?

[JoeyG1973]

Figured it out.

https://gist.github.com/JoeyG1973/69ae503f67ff7f07b498b2e53226e206

[cobusdebeer]

@kapilt Thank you! Your method works for me as well. In my case, the client application gets cognito credentials (expiring in 1 hr) and does an s3 upload that may run for hours. I used assume_role_with_web_identity() in the refresh, with WebIdentityToken = the cognito token

[Enrico2]

It's been more than a year since the last activity on this thread. Have people found a better workaround than @kapilt 's? Thanks.

[slothkong]

@BardiaAfshin, I have the same issue with S3. Any chance you found a solution?

[jappievw]

Found a solution which only uses the public api of Boto3 and Botocore. Gist can be found here. Feedback = welcome.

[sls-cat]

https://github.com/monkeysecurity/botor is really close to doing what you need. There is at least one additional incarnation of this https://github.com/Netflix-Skunkworks/cloudaux, but I found that some of them just refresh the STS token and don't cache the connections. It's not hard to test and fix. It is a python decorator that you wrap all your AWS SDK calls with. It creates tokens and refreshes them when they are almost expired. It also caches connections. I made minor modifications so that it works with or without assumed role. It's not seamless, but it it requires no changes to core AWS SDK and it is working well for our needs.

[DogFortune]

I have the same problem. In my case I would like to transfer files to S3 using AssumeRole. Since transfer stops when it expires, you need to update your credentials. Is there a better way to update my credentials without stopping the transfer?

[chrisdlangton]

I strongly suggest this issue be CLOSED - it is a non issue. i.e. chose the tool for the job. AWS solve this use case, all you need to do is use an instance profile and assign the role to it, Amazon will keep the credentials alive for you.

If you are not using a host managed by Amazon, and the 12 hour limit on STS Assume Role is too short for your use case, then STS Assume Role is not the tool for the job, you will have to do these problematic hacks to make the square peg fit in your round hole, not smart.

The appropriate solution might be using IAM keyId and secret, or federated through Cognito, or something else. Bottom line, if you need more than a 12 hour session for a long running use case STS Assume Role is an anti-pattern and should be avoided.

Note to maintainers; do not change botocore or boto3 to appease such feature requests that any AWS solution archetect, and all AWS documentation and materials discourage. Point these issues to the appropriate AWS documentation to help the educational issues, and suggest the Labs and AWS certified developer associate / professional exam materials.

[kapilt]

There are many legitimate use cases for this, please don't make assumptions and try to impose them on others without understanding their use cases. sts role assumption is needed for crossing account boundaries even from an instance with profile, and long running operations will need refresh. static iam creds are much more an anti pattern from a security perspective. I think there is a bit more support in botocore for refresh, but some assembly still required, the cleanest I've seen is this one https://github.com/boto/botocore/issues/761#issuecomment-426037853

[chrisdlangton]

incorrect, iam key and secret is not poor security practice at all. As an Information Security professional the security risks are poor key management. sts assumerole also requires iam key and secret that has permissions for sts and iam actions when assuming outside an aws managed compute environment therefore the same inherent risks apply in both cases. there is actually more risk when developers poorly implement hacks to force sts to work outside its 12 hour limit.

Also all code samples provided, and the feature request itself, do not address the cross account boundary use case, it is clearly a request to keep alive an assumed role for longer running processes than 12hr limitation. A good developer may find a solution and contribute upstream but a great developer solves the root problem for their use case.

Security risks aside, as this is a pure right-tool-for-the-job statement, and preserving the library from being compromised. As developers must seek clarity from those with knowledge before writing poor code that compromises the project. I impose nothing more than as boto maintainers to consultant AWS about this feature request, and I simply explain the facts to consider.

[ericfrederich]

I'm looking for a solution to this as well. I have a Lambda which assumes roles in other accounts.

boto3.Session() can take a profile_name but this requires ~/.aws/config/~/.aws/credentials files to work they these files don't exist in Lambda.

I think it would be enough if boto3 provided a way to configure profiles programmatically. Maybe that's already there? I haven't found it.

[ericfrederich]

Just found this.

Check out get_role_session from boto3_extensions ... (permalink at time of commenting)

This is exactly what I was mentioning. They're monkey-patching the configuration in memory instead of using a file. This is also a hack. There really should be a way to programmatically specify a profile.

[PSiAU]

We use role chaining which AWS sets an unchangeable hard session limit of 1 hour on, which makes credential refreshing necessary.

[chrisdlangton]

the only real question is; should a framework be dogmatic? should a framework transparently manage auth keep-alive for the user? should a framework circumvent the vendors own restrictions that are based on security best-practices?

The answer to all of those I hope keeps these rubbish insecure ideas out of boto3 and let the users who want to be insecure do their own hacks that should not be in a framework like boto3.

[ericfrederich]

@chrisdlangton when using an IAM role assigned to an EC2 instance this framework does exactly what you mention. It manages keep-alive for the user. It is completely transparent. Refreshing tokens using that 169.254.169.254 metadata service.

Here we're asking that the functionality be extended to support using an assumed role as well, not just the one assigned to the instance.

I would argue that

• Long running jobs on an EC2 are a valid use case and not against security best-practices.
• Long running jobs on an EC2 which assume a role are a valid use case and not against security best-practices.

If you have any links / documentation saying otherwise, I'd be happy to read them.

[chrisdlangton]

@ericfrederich why be so petty as to make it sound like it was my opinion in context to what you describe? I responded to @PSiAU commenting on the 1hr session limit, don't go generalizing my response to the 1hr limitation to make your own arguments sound better. Come up with your own value proposition to make the case to add bespoke hacks to boto3 that fit into your model of how frameworks should work and don't go generalizing other peoples responses to specific comments you didn't even make.

p.s. if you're looking for a link about the 1hr session limit it's clear you're not educated or capable of finding the knowledge (ignoring you asked me and I didn't even make the point)

[jimcarreer]

Also +1, my team is facing a similar issue and ended up using the gist @kapilt provided. Assumed roles are generally the solution AWS Support recommends when performing cross account operations, and there are scenarios you need these sessions refreshed. The fact that RefreshableCredentials exist at all speaks to the admission of the validity of that use case. Also I'd argue periodic session refreshing is more secure than hard coded AKey/SKey values. If the later is leaked once you have a major security issue until the keys are changed manually, depending on your setup that may require redeployment of software. If session details created from assumed role are leaked, you have a major security issue until they expire or you revoke them. The argument that refreshing them is some how insecure, especially considering that this is very similar to how the credentials for instance profiles are rotated, doesn't make a lot of sense to me.

[kapilt]

Well reasoned and said but in general, Don’t feed the trolls

On Wed, May 22, 2019 at 3:51 PM jimcarreer notifications@github.com wrote:

Also +1, my team is facing a similar issue and ended up using the gist @kapilt https://github.com/kapilt provided. Assumed roles are generally the solution AWS Support recommends when performing cross account operations, and there are scenarios you need these sessions refreshed. The fact that RefreshableCredentials exist as at all speaks to the admission of the validity of that use case. Also I'd argue periodic session refreshing is more secure than hard coded AKey/SKey values. If the later is leaked once you have a major security issue for until the keys are changed manually, depending on your setup that may require redeployment of software. If a session details created from assumed role are leaked, you have a major security issue until they expire or you revoke them. The argument that refreshing them is some how insecure, especially considering that this is very similar to how the credentials for instance profiles are rotated, doesn't make a lot of sense to me.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/boto/boto3/issues/443?email_source=notifications&email_token=AAAFJET7DMH74ZJYNOQ472DPWWW3FA5CNFSM4BYZZWHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWAJXAQ#issuecomment-494967682, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAFJES3LZOQU7YVFKSPCD3PWWW3FANCNFSM4BYZZWHA .

[chrisdlangton]

@kapilt so a comment that only name calls and confirmation bias callouts is called what?

On-topic, the most recent example provide here by @PSiAU has an AWS restriction of 1hr. If you do not understand the security for that choice, please do not assume that you are correct to circumvent the security controls by promoting boto3 to add your preferred solution to circumvent a security control.

I'm not arguing for imposing any restrictions on anyone, I am arguing to avoid boto3 enabling users who want to circumvent security controls.

case-in-point I have a separate feature request and possible PR to add Perfect Forward Secrecy to boto3.

Respect AWS security controls should not be circumvented. Period.

[kapilt]

@chrisdlangton, recapping

• sts session time out is configurable from 1-12 hours, it is not 1 hour, that is merely a default. https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
• static credentials for iam users are a security anti-pattern, many organizations disable iam users api keys entirely.
• control of whether or not session renewal is possible is controlled via iam permissions, ie the ability of the iam role to invoke AssumeRole on itself, not by the sdks
• attempting to prevent sdks from offering ergonomic interfaces in the name of security, when the underlying capability is available and in common use, is arguing for security through obscurity while misunderstanding the real security controls here are iam based not sdk.

there are many legitimate application use cases for this, when performing any long running operation that needs to cross role or account boundaries. the job of sdks is to enable api usage and application development, if there are common tasks for applications around api usage, rather than having everyone reimplement the same, the better outcome is to have the sdk support that functionality directly.

[olemarkus]

100% agree with @kapilt here.

[ericfrederich]

@chrisdlangton, mentioning the person you're responding to is not being petty. I was merely responding to your rhetorical questions. Without mentioning you it would have less context.

Thanks @kapilt and @jimcarreer ... you put my thoughts into words better than I can.

For now I continue to use get_role_session from boto3_extensions for two different use cases.

• it allows me to assume a role in a Lambda using a profile where ~/.aws/credentials and ~/.aws/config don't exist
• it allows me to have long running scripts use the assumed role

[chrisdlangton]

your use cases (you think they are valid and that's fine) can be implemented using boto3 right now, you don't need boto3 to provide you functionality that circumvents AWS security controls be it 1hr or 12hrs today or 9999999hrs tomorrow because these are security controls whether you like them or not.

If boto3 were to have a built in way to circumvent these, which is an insecure function by nature, boto3 would be encouraging that practice for all users whether or not they have valid uses (you call them valid, they call them valid, who cares if they are valid).

If anyone has valid reason to argue for this feature, speak to your AWS TAM or SA to get AWS to make a change to the security controls. Stop pressuring boto3 to circumvent them for you, you already have the ability to implement such circumvention on your own without compromising anyone other than yourself.

[jimcarreer]

your use cases (you think they are valid and that's fine) can be implemented using boto3 right now, you don't need boto3 to provide you functionality that circumvents AWS security controls be it 1hr or 12hrs today or 9999999hrs tomorrow because these are security controls whether you like them or not.

No one's advocating bypassing these. If you look at @kapilt gist: the maximum amount of time the session lives is still whatever the maximum session time assigned to the role, ie a hard max of 12 hours. We're advocating for a non-hacked way to replace that session using the RefreshableCredentials class that's already provided by botocore. It would probably be better to call this "session rotation" not "session refreshing" because that's what it is: when the session is expired, the gist that @kapilt creates a new one, the old one is still invalid and still expires as it should, the new one seamlessly replaces it within the boto internals. The new assumed role session is created via the instance profile session which you already suggested people should be using. This type of rotation in the boto internals is exactly how regular sessions based on the instance profile are rotated and why when performing long running aws cli operations on an IAM you don't receive a 403 mid completion for the originating session expiring. Literally everyone in this thread wants to extend that behavior to allow an assumed role session to do the same thing.

[chrisdlangton]

@jimcarreer per the OP

boto3 looks like it wires in credential refreshing when its using instance roles by default. The issue is how how to setup the same behavior when doing sts role assumption, to refresh credentials (i'm initially starting with an instance role). Atm the only way to pass the sts assumed role credentials is in via explicitly creating a session/client with key/secret/token, but that results in the creation of a default Credentials class without the refresh behavior. There doesn't seem to be a clean way to provide a RefreshableCredential acquired with sts token assumption to a client.

refreshing enough said?

I was going to point out the whole distinction without a difference, but given you missed the OP and have clearly your own ideas of the feature request - perhaps rather you might have a better day if you start your own feature request for rotate and try to make it sound useful. Good luck

[jimcarreer]

@chrisdlangton

I was going to point out the whole distinction without a difference

I firmly believed your disagreement came from an initial misunderstanding of the more technical mechanics of the request, I wasn't attempting to describe a difference because, as you point out, there isn't one. I was simply trying to rephrase the request in a way you might better understand. Your statements here show that you obviously don't disagree with the automatic credential refreshing that occurs when using an instance profile in boto, so it is hard to imagine why you'd disagree with this behavior being extended to a session using an assumed role, unless it was unclear to you how this could be done in a secure manner.

I still believe you don't quite understand the request or the details of the suggested implementation, but you also don't seem to be willing to discuss the issue without being patronizing to the people involved. I don't feel engaging with you further would lead to any fruitful outcomes. Apologies that I could not better describe the request here.

@kapilt thanks again for the gist, our security team was extremely happy we came to a solution that did not require a permanent key-pair (these are only granted in very special cases), which was really the only other option for us.

[chrisdlangton]

@jimcarreer I'm glad you found the simple hack provided by @kapilt useful - and I am concerned that you and others feel boto3 should be providing circumvention for users to avoid AWS security controls.

Until now I have stayed on topic, sts assume role refreshing, but it is clear to me now that you feel strongly that precedence of boto3 providing code to work within the instance profile scenario should also give magical inherent approval to sts assume role refreshing also. This is simply ludicrous, there is no such precedence. For precedence to exist the 2 scenarios must be serving the same purpose and goals. if the goals of sts and instance profiles were identical the security controls would reflect this - the fact AWS do not apply the same security controls to each clearly tells you there is no precedence to reference for this feature request.

The best advice I can give anyone who has a problem with AWS security controls is, speak to AWS about changing the security controls - alternatively you have the option to work-around the controls (using the hacks provided in this thread).

Boto3 is a freamework, it has design goals to provide an SDK to support the AWS API. There is no boto3 design goals that support circumvention of AWS security controls.

[ronaldozark]

@kapilt I opened a PR to allow botocore credentials to be set directly for a session, I had the exact same issue as you did and basically arrived at the same solution. On my travels I found a helper function in botocore:

https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L148

def create_assume_role_refresher(client, params):
    def refresh():
        response = client.assume_role(**params)
        credentials = response['Credentials']
        # We need to normalize the credential names to
        # the values expected by the refresh creds.
        return {
            'access_key': credentials['AccessKeyId'],
            'secret_key': credentials['SecretAccessKey'],
            'token': credentials['SessionToken'],
            'expiry_time': _serialize_if_needed(credentials['Expiration']),
        }
    return refresh

Which looks like it encapsulates at least part of your gist (including the silly serialization of the date).

[chrisdlangton]

@ronaldozark +1 for create_assume_role_refresher not being in boto3 Let us find out how botocore feel about the code lacking support for MFA, at least the error can be handled by user space code.

[ronaldozark]

Let us find out how botocore feel about the code lacking support for MFA

Not sure how MFA is relevant here? Botocore basically supports this assumed role session refreshing you just need to be able to set the credentials object in the session.

[chrisdlangton]

Without intending to derail the on topic conversation, MFA is an important security feature that is commonly used for an IAM Role. There are many times a human is responsible for providing the 6 digit code but in applications that use MFA-enabled roles the 6 digit code is derived from a secret key provided to the application using any form of secrets management tool such as Parameter Store or Hashicorp Vault which I'm familiar with.

@ronaldozark I applaud you making upstream contributions, you have good initiative and I hope you receive support and direction from botocore, however this is boto3 and the feature request itself you have already made clear has nothing to do with your botocore utility function so lets stay on topic here please.

[ronaldozark]

Without intending to derail the on topic conversation, MFA is an important security feature that is commonly used for an IAM Role.

Right, everyone knows what it is, it just odd you mentioned it here as:

1) It never came up in any previous discussion on the issue. 2) It has very little with whats being discussed here.

It seems odd that you would mention it? It's like if everyone was discussing a car's alternator and you brought up the head lights? Just doesn't make a lot of sense.

[chrisdlangton]

I've been consistent;

• I care about AWS security controls
• MFA is a security control
• you provided code in this thread that calls client.assume_role()
• client.assume_role() prompts for MFA
• Your code does not handle MFA prompt, it errors to user space as I pointed out earlier, and the challenge can never be fulfilled in your code.
• again, MFA is a security control, i care about AWS security controls
• as i said, I hope you receive support and direction from botocore

Let's see how boto3 maintainers feel about this feature request, we have certainly exhausted the discussion on the purpose of the feature and i'm comfortable with the message of maintaining AWS security controls. For both sides of the discussion I hope boto3 maintainers can provide guidance to those who want the capability while respecting AWS security controls.

[ronaldozark]

client.assume_role() prompts for MFA

Only if configured to.

[ronaldozark]

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role

(Optional) You can include multi-factor authentication (MFA) information when you call AssumeRole . This is useful for cross-account scenarios to ensure that the user that assumes the role has been authenticated with an AWS MFA device. In that scenario, the trust policy of the role being assumed includes a condition that tests for MFA authentication. If the caller does not include valid MFA information, the request to assume the role is denied. The condition in a trust policy that tests for MFA authentication might look like the following example.

This has to be true:

In that scenario, the trust policy of the role being assumed includes a condition that tests for MFA authentication.

Otherwise there is no MFA handling required. This is not true for my scenario, nor is MFA a mandatory requirement for assuming roles, even across two AWS accounts. Its pretty easy to verify this yourself.

[chrisdlangton]

I agree that MFA is optional, IAM Roles are optional too right? You're clearly missing the point..

Only if configured to

botocore and boto3 are frameworks, code contributions must handle all known configurations, anything less is a defect at best, let's leave the botocore discussion in botocore please.

[ronaldozark]

botocore and boto3 code contributions must handle all possible configuration

Ah but you see it does! If MFA were enabled in the scenario described, it would have to be handled in the custom implementation of the refresh(...) function provided by the user. So it is supported, but if a user is already trying to leverage the RefreshableCredentials object they'd have to handle the MFA prompt in the refresh() call which is implemented by the API user in this case. Otherwise, you'd be denied access.

You're clearly missing the point.

Ah, I think its more we might be have different use cases in mind and are talking past each other! But hey thats how we figure things out.

[ronaldozark]

Taking the OP's gist as an example:

def refresh():
        credentials = session.client('sts').assume_role(
            RoleArn=role_arn,
            RoleSessionName=session_name)['Credentials']
        return dict(
            access_key=credentials['AccessKeyId'],
            secret_key=credentials['SecretAccessKey'],
            token=credentials['SessionToken'],
            # Silly that we basically stringify so it can be parsed again                                                                                                                                                                                                                                                                              
            expiry_time=credentials['Expiration'].isoformat())

Basically if you had MFA enabled for the role specified by role_arn, then it would be up to the implementer to provide that MFA code to the assume_role(...) call. Its handled at the level where the API user is implementing their code, its just the OP's example is for roles with out MFA enabled.

But that is a good point! Thanks for calling it out.

[chrisdlangton]

it would have to be handled in the custom implementation of the refresh(...) function provided by the user

Your PR defines said function, which in turn prevents the user from defining as you suggest

I feel you have grasped the catch22 now, your proposed botocore internal is now performing work intended for botocore user land - and this version is lacking a security control the AWS Well Architected Framework, CIS benchmarks, PCI DSS 3.2, FedRAMP, HIPPA, (list goes on) all require.

Bottom line, both this boto3 thread and botocore PR are lacking some fundamental security considerations, I am happy to connect with you (or really anyone) and provide more details or even guide you on your efforts if you want to apply a degree of security controls to your projects - i do this for the love of it and I just happen to be lucky enough to do what i love for work too.

Let's wrap this up and focus the thread on boto3 issue 443 solely

[ronaldozark]

Your PR defines said function, which in turn prevents the user from defining as you suggest

It does not actually. Please point to the code in my PR that implements a refresh function.

[ronaldozark]

Oh maybe you're confused. If you're talking about this:

def create_assume_role_refresher(client, params):
    def refresh():
        response = client.assume_role(**params)
        credentials = response['Credentials']
        # We need to normalize the credential names to
        # the values expected by the refresh creds.
        return {
            'access_key': credentials['AccessKeyId'],
            'secret_key': credentials['SecretAccessKey'],
            'token': credentials['SessionToken'],
            'expiry_time': _serialize_if_needed(credentials['Expiration']),
        }
    return refresh

^ This is not my code or in my PR. This already exists in botocore today, actually its been in botocore since about 2015: https://github.com/boto/botocore/commit/2dae76f52ae63db3304b5933730ea5efaaaf2bfc

I can see how that might have confused you. My PR simply allows someone to set the Credentials object on a session directly, it leaves the implementation refresh(...) to the user (where it should be).