Support for Signed Requests / AWS Elasticsearch

[csepulv]

For services like AWS Elasticsearch, all requests needed to be signed and the signature added to the headers (at least for some configurations in AWS).

Is there any way to specify custom headers on a per request base? While I still have a proxy for my requests, I am depending on AWS to verify security, which works fine if I can sign the request.

Thanks

[siddharthlatest]

@csepulv You can pass custom headers to the ReactiveBase component https://opensource.appbase.io/reactive-manual/getting-started/reactivebase.html#usage.

[csepulv]

How would I update the headers on a per request basis? I need to sign each individual request.

Thanks

[siddharthlatest]

@csepulv We will probably need a way to allow passing custom headers per request. I am leaving this open to get some suggestions, cc @metagrover @divyanshu013.

[csepulv]

FWIW, I went spelunking in the codebase and perhaps a hook in appbase/lib/fetch_request.js, which would be exposed possibly as an attribute on ReactiveBase.

I imagine appbase/fetch_request.js needs to be involved as I would need access to the pending request to generate a signature.

Unfortunately this would involve updating the two projects (appbase-js and reactivesearch), but hopefully it is a reasonable change as request signing is used in various approaches to securing APIs.

[davidklebanoff]

You could add the desired headers in the proxy service.

On Fri, Jun 22, 2018 at 12:27 AM, Christian Sepulveda < notifications@github.com> wrote:

FWIW, I went spelunking in the codebase and perhaps a hook in appbase/lib/fetch_request.js https://github.com/appbaseio/appbase-js/blob/master/lib/fetch_request.js, which would be exposed possibly as an attribute on ReactiveBase.

I imagine appbase/fetch_request.js needs to be involved as I would need access to the pending request to generate a signature.

Unfortunately this would involve updating the two projects (appbase-js and reactivesearch), but hopefully it is a reasonable change as request signing is used in various approaches to securing APIs.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/appbaseio/reactivesearch/issues/419#issuecomment-399317023, or mute the thread https://github.com/notifications/unsubscribe-auth/AHgKwja1VGsBMKwKyNUs-zCM3vMlqIK0ks5t_HIbgaJpZM4Uy0vU .

--

David Klebanoff Aventrix | Founder & Engineer | www.aventrix.com

[siddharthlatest]

That's a great point. @csepulv are you already using a middleware proxy?

[davidklebanoff]

@siddharthlatest He stated he does use a proxy in his original post.

Using client-side signing means clients would have access to your AWS Access Key, and thus would be able to sign any request they want. I fail to see how this increases security, if anything it just exposes your Access Key to the world.

[csepulv]

I am using a proxy. I have Amazon Elasticsearch behind a proxy. The proxy is fronted by AWS API Gateway.

A user signs in via AWS Cognito and is given temporary / session credentials. These credentials are used to sign requests and authenticate calls to the API Gateway. AWS Cognito and API Gateway handle the signature and credentials verification (i.e. security), if the request is signed.

For my other requests, this works fine. I can sign the API requests with the temporary credentials the user has for the session. But for the Reactivesearch, I don't have a way to intercept the individual request and add the signature headers.

I didn't go into the full details, as I was trying to simplify my request, which is for the ability to each sign individual request.

[siddharthlatest]

You can in fact connect to Elasticsearch with ReactiveSearch via a middleware proxy - https://opensource.appbase.io/reactive-manual/getting-started/reactivebase.html#connect-to-elasticsearch.

We have a sample proxy server written as well - https://github.com/appbaseio-apps/reactivesearch-proxy-server, I think the same principle should work when using with API Gateway.

[csepulv]

Thanks for the links. I had made use of them as I built my proxy. My proxy works if I shut off authentication (or use a mechanism like an API key in the header). But if I enable the AWS Cognito / API Gateway integration, all requests fail as a result of the lack of request signature. (As noted, the other API requests I can control work fine.)

I am not trying to start a security debate, as I think different approaches have their own benefits and tradeoffs. But it seems that Reactivesearch only supports basic auth or some other "static" header mechanism, such as an API key. (I write "static" in the sense that the headers are consistent across some extended lifecycle, such as the object life time of a ReactBase instance.)

I am hoping you might consider providing support for security models that require individual request signing.

[siddharthlatest]

@csepulv While I am not familiar with the specifics of AWS services you are using, I am still curious on why the middleware can't handle this (i.e. signing requests) if it can be handled by adding a dynamic header into the ReactiveSearch library itself. Since each request first passes through the middleware, wouldn't it be just as easy to hook in the headers before sending them to the AWS backend?

That said, I am not against including this feature into the library at some point as there are other potential use-cases beyond this, and also happy to take in PRs for the same.

[csepulv]

How would you suggest authenticating/securing the middleware? I can sign requests after the middleware (which I am already doing to get to the actual elastic cluster), but it's at the middleware I currently need signed requests?

Or are you suggesting that the middleware doesn't require/use signed requests?

[siddharthlatest]

May be I am missing how your flow works .. Is it ReactiveSearch -> Middleware -> Cognito / API Gateway? Or some other way?

As long as middleware proxy routes to other services, you can sign the requests via it. Any docs / blog post I can read up on the general flow? Just curious to understand if we are missing out something important.

[csepulv]

The flow is Reactivesearch -> API Gateway -> Elasticsearch

The API Gateway is the entry point for the middleware/proxy (lambda function).

That said, I am not against including this feature into the library at some point as there are other potential use-cases beyond this

As far as other user cases, I think request signing is a relatively common security mechanism. (I've used and written numerous APIs that use it.) While my need is AWS related, I think any middleware/proxy that uses request signing would need a change in Reactivebase to support it.

I will fork Reactivebase, etc. and send a link to the repos. I will do what is expedient, as I don't know the Reactivebase/Appbase architecture well enough to anticipate how you might prefer to implement it. But it will show one approach at least.

[siddharthlatest]

Is it fair to say that there is a secured endpoint which using some identification mechanism (OAuth, et al) signs the request which is then sent to your API gateway? And middleware can't do this instead because when the request is in transit, you can no longer authenticate. (edit: still slightly counter intuitive as you typically get an access_token with an OAuth like approach which can be sent to the middleware to then sign it)

Looking forward to seeing the approach 👍, as regardless - having the ability to modify the network request that is sent can be helpful in some scenarios.

One way I can think of us supporting this is as a callback function prop of ReactiveBase component, and has the ability to update uri, method, headers and body.

beforeSend={(uri, method, headers, body) => {
  // make changes
  ({  uri: updated_uri,
      method: updated_method,
      headers: updated_headers,
      body: updated_body })
}}

This way, a user has full freedom over what gets sent in each request. Implementing this prop is optional (obviously) and should only be done if you are looking to modify the network request defaults that the library provides.

We will need some changes in appbase-js to accept new headers for each request.

[csepulv]

I've created a fork and made the minimal changes to support request signing. The changes to appbase are here https://github.com/csepulv/appbase-js/commit/88d5d5af6f2c152b35d69a4b255b048a05a36c09

And the changes to reactivesearch are here https://github.com/csepulv/reactivesearch/commit/825e423964cdf4173741df761fab01edeaf895c9

I've confirmed this works for my case. Here is an example os usage (in the component that renders ReactiveBase)

 signRequest = (request) => {
    const hostUrl = 'https://my-host';
    const host = 'my-host';
    const app = 'my-app-name';

    const credentials = this.state.credentials;
    if (credentials) {
      const { accessKeyId, secretAccessKey, sessionToken } = credentials;
      const opts = {
        method: 'POST',
        service: 'execute-api',
        region: 'us-west-2',
        path: `/${app}/_msearch`,
        host: host,
        body: request.body,
        headers: request.headers,
        url: `${hostUrl}/${app}/_msearch`
      };
      const signedRequest = aws4.sign(opts, { accessKeyId, secretAccessKey, sessionToken });
      delete signedRequest.headers.Host;
      delete signedRequest.headers['Content-Length'];

      //aws4.sign mutates the headers, adding the signature headers
    }
  };

  render() {
    console.log(`auth:${this.state.authenticated}`);
    return (
      <Container>
        <ReactiveBase
          beforeSend={this.signRequest}

This mutates the request. An alternate would be to restrict changes to the headers only, and provide beforeSend (or whatever you want to call the handler) a copy of the request and have it return headers that get merged onto the original.

Let me know where you'd like to go from here.

[metagrover]

@csepulv This looks good to me. I believe signRequest will return the updated request here right?

If that's the case, then can you please send in a PR for this on appbase-js and reactivesearch?

Also, include beforeSend in the propTypes of ReactiveBase 🐱

[csepulv]

The beforeSign method, (signRequest in this case), mutates the request (including the headers).

I added the propType and have submitted the pull request. Let me know if there is anything else to change.

[metagrover]

We definitely don't want to mutate the request object here, so if you can update it to work in an immutable way, that'd be perfect.

We ideally want to receive a request object in the beforeSend function and expect it to return an updated request (newly created) object.

---

Since it can have nested levels of objects, we can simply do:

const newRequest = JSON.parse(JSON.stringify(request))

to copy the object contents of the request object in the immutable way.

[csepulv]

Would you prefer that fetchRequest does the copy, to protect against the beforeSend handler possibly mutating the request? something like...

if(this.client.beforeSend){
            const requestCopy = JSON.parse(JSON.stringify(requestOptions));
            const newRequest = this.client.beforeSend(requestCopy);
            if(newRequest)
                requestOptions=newRequest;
        }

But since requestOptions is a const, either it needs to be declared as a let or do something like

if(this.client.beforeSend){
            const requestCopy = JSON.parse(JSON.stringify(requestOptions));
            const newRequest = this.client.beforeSend(requestCopy);
            Object.assign(requestOptions, newRequest);
        }

preference? something else?

[csepulv]

Also, I used the name beforeSend as i took it from an earlier comment. It might be better to rename it to something like updateRequest or buildUpdatedRequest.

But any of it is fine. I can change the PR as needed

[csepulv]

any thoughts on the PRs?

[metagrover]

Hi @csepulv, sorry about the delay on this. It doesn't make much sense for the library to send the given function a request clone which can be modified. It'd be best if the beforeSend function returns an updated request (newly created) object.

This way, we can ensure that the immutability practise is being followed - which will be easier on the eyes of the reader allowing him/her to trace the changes on the request object whenever needed.

[csepulv]

I've updated the PR with this commit, so that if a new request is provided by the handler, it is used for the fetch.

[metagrover]

Looks good overall. Will do some minor cleanup and merge in sometime. Thanks!

[csepulv]

Thanks for merging the pull requests. I will close this issue.

[csepulv]

I've been using the transformRequest prop/method to sign AWS requests for a while. (It used to be called beforeSend in earlier version.)

https://opensource.appbase.io/reactive-manual/getting-started/reactivebase.html documents the prop.

I use aws4 to sign the request going to AWS API Gateway / Lambda proxy for the ES instance.

On the AWS API Gateway, I assume you have a Lambda that actually connects to ES. Is the AWS ES cluster is in a VPC, you will need to provide those details to the Lambda, else it can't reach it. But that may not at all be related.

FWIW, I would use Postman to verify you can use your AWS API Gateway to make ES requests. It supports AWS S4 signatures. If that doesn't work, any issue is separate from Reactivesearch

[jake-ingram]

@csepulv Would you mind sharing some examples for the configuration in both reactivesearch and the Lambda proxy? I'm having trouble visualizing the specifics of the entire process.

[csepulv]

Unfortunately, I don't have code I could post publicly. However, I once wrote a blog post explaining the general auth process.

https://medium.com/free-code-camp/how-to-secure-microservices-on-aws-with-cognito-api-gateway-and-lambda-4bfaa7a6583c

On the Reactivesearch side, set a callback via transformRequest. In the callback, sign requests with aws4.

https://github.com/csepulv/auth-api-demo/blob/master/web-ui/src/api-client.js is an example of signing such a request and this method would be the base of your transformRequest handler.

On the AWS side, in the example in the blog post, https://github.com/csepulv/auth-api-demo/blob/master/aws-api/api.js uses the API gateway to check auth. The API lambda could then use an ES client (or direct HTTP) to call the Elasticsearch cluster. (If the ES cluster is in a VPC, you will need extra config.)

I hope this helps.