SigV4 sign arbitrary requests

[rhboyd]

Botocore Sigv4 signs all AWS API Requests. It should expose the ability to sign arbitrary https requests.

API Gateway has an AWS_IAM auth mechanism and this results in needed to sigv4 sign https requests to domains not included in the usual Python SDK. As more databases (like Neptune) offer AWS IAM auth to manage database permissions, this problem will appear more frequently. The current "recommended" approach is to use a random third-party library named "aws-requests-auth" but that library isn't maintained by AWS and I'd really like to avoid asking new devs to Google "AWS SIgv4 Python", which returns about a dozen different repos. Should AWS be encouraging people to use 3P libraries for something as important as signing requests?

I propose that botocore expose some functionality to attach SigV4 Auth to any request. I think this can be accomplished by adding a def __call__(self, request) method to the SigV4Auth class and I'd even be willing to implement and test this feature if the botocore team agrees that this feature is needed.

I hacked together a way to perform the signing with the existing functionality, but it's pretty sloppy and requires building two requests in parallel then stripping the auth bits out of one so that we can send the other.

https://gist.github.com/rhboyd/1e01190a6b27ca4ba817bf272d5a5f9a

[benkehoe]

In my ideal world, this is available on the Session object, so I could do something like session = botocore.Session(); response = requests.get(url, auth=session.signer(service='execute-api')) or something similar. And then also made available on the boto3 Session object, so I don't have to muck about getting the botocore session out of it.

[swetashre]

@rhboyd -Thank you for your post. I would mark this as a feature request.

[tyldavis]

I was surprised to find this wasn't already a feature! This would be hugely beneficial for AWS_IAM secured API Gateway endpoints.

Edit: I am especially surprised after finding this is a feature of the Go SDK, and in fact it appears all other SDKs support this already.

[ryansb]

I'd also be happy to do implementation work on this, as long so the botocore team is ok with the new __call__ on SigV4Auth.

[benkehoe]

It could be a new wrapper class with a __call__ method

[richardhboyd]

I've discovered a cleaner way to do this lately

import boto3
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import requests

session = boto3.Session()
credentials = session.get_credentials()
creds = credentials.get_frozen_credentials()

def signed_request(method, url, data=None, params=None, headers=None):
    request = AWSRequest(method=method, url=url, data=data, params=params, headers=headers)
    # "service_name" is generally "execute-api" for signing API Gateway requests
    SigV4Auth(creds, "service_name", REGION).add_auth(request)
    return requests.request(method=method, url=url, headers=dict(request.headers), data=data)

def main():
    url = f"my.url.example.com/path"
    data = {"environmentId": self._environment_id}
    headers = {'Content-Type': 'application/x-amz-json-1.1'}
    response = signed_request(method='POST', url=url, data=data, headers=headers)

if __name__ == "__main__":
    main()

[tyldavis]

I've discovered a cleaner way to do this lately

[...]

Isn't this basically the same thing as your original Gist (I'm assuming @rhboyd and @richardhboyd are the same person... apologies if not), but without calling AWSRequest.prepare()? Looking at the source it seems like it isn't doing much anyway.

[richardhboyd]

yeah, the request.headers part can also be re-used for websockets that need IAM Auth (Neptune Gremlin connections) and the credentials work in a more environment agnostic way as well.

[RahulPATK]

@rhboyd can we also pass security token as well? I need to pass assume role credentials along with signed header?

[richardhboyd]

can you clarify the question a little bit? you can sign requests using the security token as well but you shouldn't have to pass the actual credentials in the header. the credentials should never be sent with a request (only a signature made from the credentials should be sent)

[RahulPATK]

@richardhboyd thanks for coming back, so when I tried your botocore with AWSRequest and Sig4Vuth approach I was getting that 'code': 'ACCESS_DENIED', 'message': 'User: arn:aws:sts::myrole is not authorized to perform: execute-api:Invoke on resource: soyrcerole /GET/sites with an explicit deny'}

I tried second approach where I manually signed the signature using aws documentation(https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html) its saying security token is invalid 'message': 'The security token included in the request is invalid, so I thought to pass security token and provided signed again, I was getting INVALID_REQUEST The request signature we calculated does not match the signature you provided.

Checkyour AWSSecretAccessKey and signingmethod.Consult the service documentation for details.TheCanonical String for this request should have been. and just for FYI, its working on POSTMAN when I'm passing assume role credns.

[richardhboyd]

the first message meant that the request was signed properly but you didn't have permission to invoke the API. what does the API's resource policy look like?

[RahulPATK]

I confronted same to use but they have added my lambda exn role in there config, and api resource policy expects credentials need to be added to a header in the request somehow. But the kind of header the request needs is to do with sigv4 signing

[richardhboyd]

Okay, so there's a few different resources here we need to be clear about.

-API Gateway API -API Gateway execution Role (let's call this RoleA) -Lambda Function Execution Role (RoleB) -Role the signs requests that are sent to the API (RoleC)

The API Gateway resource policy needs to allow RoleC to "execute-api" for any path in the API that is needed.

RoleA needs permission to invoke the Lambda Function and needs to have a trust policy that lets the API Gateway service assume that role.

On Nov 17, 2021, at 8:14 AM, RahulPATK @.***> wrote:



I confronted same to use but they have added my lambda exn role in there config, and api resource policy expects credentials need to be added to a header in the request somehow. But the kind of header the request needs is to do with sigv4 signing

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/boto/botocore/issues/1784#issuecomment-971568134, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4IKXY6PNMYNKNH3S5NDT3UMOTAJANCNFSM4ID37Q5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[RahulPATK]

-API Gateway API -API Gateway execution Role (let's call this RoleA) -- Configured -Lambda Function Execution Role (RoleB) -- Configured -Role the signs requests that are sent to the API (RoleC)

The API Gateway resource policy needs to allow RoleC to "execute-api" for any path in the API that is needed -- need to check

RoleA needs permission to invoke the Lambda Function and needs to have a trust policy that lets the API Gateway service assume that role -- Configured

[RahulPATK]

@richardhboyd its done, issue was with headers,

so, first, I manually signed, using aws documentation, https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html, second, please make sure to add security token when using assume role third, please make sure canonical String will in appropriate order

bingo.