AWS request signing support

[rimutaka]

AWS runs its own distribution of ES called ElasticSearch Service, which is mostly compatible with this library, except for the missing request signing feature. It makes impossible to use this library with IAM-based access control.

I totally understand your unwillingness to support their distro as stated in https://discuss.elastic.co/t/any-way-to-sign-requests-to-aws-elasticsearch-using-asp-net-nest/245938, but I think this strategy is counter-productive. Consider my case ...

I am building an app prototype on AWS. ES is part of my stack. I need to get everything up and running fast and then switch to a properly supported AWS set up by Elastic. I started with AWS ES Service ad would like to use this client lib to make it easier to switch to Elastic later, but I can't. So I am ending up with a lot of custom-built code to talk to the ES instance which only entrenches me deeper in AWS services.

Possible solutions:

1. Add AWS request signing
2. Allow intercepting the request to add our own headers after it was constructed (could be an even more flexible solution)

I'm happy to put in some effort into a PR if you are interested.

Background info:

• AWS ES access control
• Signing process
• Rust signature implementation

[treykasada]

Just hit this as well. Would love to see some support here.

A possible approach could be to redefine Transport as a trait, allowing developers to specify their own implementation for the send method. It should then be possible to create a simple wrapper over one of the builtin transports that could leverage something like rusoto_signature to do the signing.

[russcam]

Thanks for opening @rimutaka.

Whilst the client won't ship with AWS request signing as part of the package, its architecture should make it possible for a user to extend the client to support this.

I am building an app prototype on AWS. ES is part of my stack. I need to get everything up and running fast and then switch to a properly supported AWS set up by Elastic. I started with AWS ES Service ad would like to use this client lib to make it easier to switch to Elastic later, but I can't. So I am ending up with a lot of custom-built code to talk to the ES instance which only entrenches me deeper in AWS services.

Genuine question- Have you considered using Elastic Cloud instead of AWS Elasticsearch? It's super easy to spin up a cluster on Elasticsearch service on Elastic Cloud, running on AWS, be billed for it through AWS Marketplace, and configure private link to allow communication over internal networks. You'd also be able to use a bunch of different credentials such as tokens, API keys (and SAML or OpenId Connect SSO on higher tiers).

A possible approach could be to redefine Transport as a trait

My intention originally was to make Transport a trait 🙂 One problem with this right now is that traits cannot have async functions, so defining send() on a pub trait Transport is not possible without using the async-trait crate. I would like to better understand the ramifications in using async-trait, including performance, before committing to any change though.

Another approach could be to expose a way to set a function that can be called just before the request is made, which I think is what @rimutaka is proposing.

[rimutaka]

I agree that async-trait is probably an overkill just for this narrow use case. An optional signing function is an easier solution. The signing code doesn't need to be part of this crate. It can come from rusoto-signature and be added as an example.

[treykasada]

@russcam If it helps, rusoto makes fairly liberal use of that crate, so it some semblance of precedent there.

To me, this kind of functionality seems almost exactly like what Transport is conceptually for (in terms of where it sits in the architecture), but honestly, either approach would work. 🙂

[meghashyam6]

@rimutaka have you been able to use this crate to connect to AWS ES eventually?