requests-compatible signer

[benkehoe]

This is a long-standing feature request of botocore: https://github.com/boto/botocore/issues/1784

Both botocore and this project use custom HTTP request representation, and the signing mechanism works against this custom representation. This makes it difficult to SigV4 sign an arbitrary request using the most-commonly and -recommended HTTP library requests, which is necessary when using API Gateway with IAM authentication.

The CRT bindings should provide a function compatible with the requests auth interface (that is, it takes a requests.Request and adds the SigV4 headers). This function could then be exposed by higher-level interfaces (e.g., boto3) without needing to translate a requests.Request into the CRT's custom HTTP request representation. I think this should be doable without adding a dependency on requests.

[nateprewitt]

Hey @benkehoe, thanks for the feature request! I agree this should be fairly straight forward from Requests perspective, we just need to generate an arbitrary callable that produces headers from the CRT. Whether we can do this without the CRT C code converting to its internal request representation is a bit more difficult. I'll follow up with the CRT team on options.

In the short term, we've already done something adjacent in the Amazon Transcribe Streaming SDK that's doing this for a Requests-like PreparedRequest object. How would you feel about moving this out into a more generalized interface if the conversion is an acceptable performance hit?

I will note we have plans underway for stand-alone Python signers in the near future as well, but we don't have a timeline to share yet.

[benkehoe]

I think this library could provide the interface backed by the conversion step as a starting point, that could be optimized later?

It seems like the root of the issue is that the signing functions only ever modify the request in place, rather than having a function that returns the additional signing headers.

[graebm]

This is possible, the underlying C libraries that do signing actually work as you describe: An abstract request can be provided, and the additional signing headers are returned, so that you can apply the transformation yourself.

When binding that C code to python we took a shortcut and just had it take an awscrt.http.HttpRequest and modify it in place, because that was simple and all we needed at the time.

But it's still fully possible to bind the more abstract API. Just need time to do it. I can't make promises, but we have plans to touch this API again in the near future, hopefully we could slot this in while we're in there.

[nateprewitt]

Hi @benkehoe,

This isn't exactly what you requested, but I wanted to let you know about a recent experiment we've made public, aws-sdk-python-signers. It's a fairly low-level library for integrating AWS SigV4 signatures into other tools. I have some samples in there under the /examples directory that will currently let you integrate with AIOHTTP, Curl, and Requests. Note, this project doesn't use the CRT.

Right now it's very much in a alpha/experimental state but it should be able to round trip signed requests to AWS in most cases. We're primarily looking for initial design feedback, any pain points encountered, and potential missing features. This is meant to be a minimal implementation, so it doesn't include things like endpoint resolution, credential fetching, etc. that we'd normally provide in the SDK.

I'll be opening an issue in that repo sometime in the coming weeks outlining more specifically what we're looking for, and how we plan to facilitate discussions, but I wanted to put it on your radar early if you have any thoughts.

[benkehoe]

Thanks @nateprewitt! Looks pretty interesting. As you might expect, I've got lots of thoughts 😁 I'll open an issue over there to talk about it.