Exception on signing PUT requests using AwsRequestSigningApacheV5Interceptor

[vahidiazar]

I am trying to sign (AWS Sig v4) opensearch requests using this library. The client library I'm using is opensearch java client and in order to use http5, I'm using ApacheHttpClient5Transport class. This is how I create a bean of AwsRequestSigningApacheV5Interceptor

    @Bean
    @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) 
    AwsRequestSigningApacheV5Interceptor awsRequestSigningApacheV5Interceptor(@Value("${AWS_REGION}") String awsRegion) {
        return new AwsRequestSigningApacheV5Interceptor("es", Aws4Signer.create(), DefaultCredentialsProvider.create(), Region.of(awsRegion));
    }

Here is how I use the interceptor:

    @Bean
    public ApacheHttpClient5Transport apacheHttpClient5Transport(Configuration cfg, AwsRequestSigningApacheV5Interceptor requestInterceptor) throws URISyntaxException {
        return ApacheHttpClient5TransportBuilder.builder(org.apache.hc.core5.http.HttpHost.create(new URI(cfg.serverUri)))
                .setHttpClientConfigCallback(clientBuilder -> clientBuilder
                        .addRequestInterceptorLast(requestInterceptor)
                        .setConnectionManager(PoolingAsyncClientConnectionManagerBuilder.create()
                                .setDefaultTlsConfig(TlsConfig.custom().setVersionPolicy(FORCE_HTTP_1).build())
                                .build())
                )
                .build();
    }

When my applications start, GET requests works as expected, but I see the following error on PUT request:

org.opensearch.client.transport.httpclient5.ResponseException: method [PUT], host [*****], URI [*****], status line [HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided.
Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'PUT
*****
accept:application/json; charset=UTF-8
content-length:47
content-type:application/json; charset=UTF-8
host:*****
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240312T030211Z
x-amz-security-token:IQoJb3JpZ2luX2VjEDMaDmFwLXNvdXRoZWFzdC0yIkcwRQIgGzc/Is/bMwbyz1qGxwgHINfduFEuDhsNSwbweO+9gd4CIQD6MjY/9FbaUEySm4MEZgqgOnUWvielLFRfJPGytopEtSrKBQg8EAAaDDkzNjY2NDQ2Mzg1NiIMNuVyrsSnflj7w9TyKqcFJwp4G4w7mETv0KS6zjtJWOwXexZaK2Ho9XbuJzSUioO4yYrQKEzAIKaDJlrbDihjBbFgp//oq7/L2EkgvGrOXoKcSvl5xCTERnyo2urX/ImyzGC9RmvzbYt4rQaKzQRssZGP5TwGs1XPDpIcu1iVnUsfu0xNZG2YhtOuLardM7yQcn5Jqt8FXOR4NZZlzjBsa/FfcpymwdbdS0TrKPpTG24s8H84HyfoSMYlPk1PpgMt1ZVJWiKRsKWquesLcpGCiF6utK++L8cimFAsEf4LxEZ4ST/lrdq8unupDN+Edm862yiC96u1FB/GdASwWHaKt3285Vau3Ce/ipjFzUEpXcs3Rd6itcS3OOIU5IrQiqgXYxtBLLbFliztDxQ+CeLvAstOB7oh2i8tyi+5dDH2ZRX5JeR0NitNJQF3plpe5ktuXjtVqZn1Wwpk4UmykJPuVfUK+CKCIebMScma0JOa589JHB/ngWbkHVZCHzIEuDrwkOK2mA7mbCXEjChg8rV244CoWg6XwEGk/+8KowL8L2CL66HJx0CLkLZ6opwmnrUlmwlsSPrbbPqU95zVQ4aOKSLjt6w7/kQTUq0TKTh0fWoPuhuU8axHfUv3JU4NUNzvyocJW16F6iajrZ+mcne5a4/0Y3MNAE4Ch4dDrKGkonWqClOeXPuttvKS0JlTI3qRbyerTMYweUPUFKmmrA6IKlog5a9l4U7xUu84l1Vt0redHPyNyImSiyWT9f73PaYEDYyGCBRFFUe9MVWgE5jSxEJ3M6Q4CvhKBfX0mwQBHXiVBBcCObCkRNAWbGORNr1++RjA5hoDEEuyEw0napxgIDawHSW5tp3rJzo5TMFzz/T1/5hVXsQKQS8uEBYk5jzBgdJqK6My4WLVOjq8L8hOjocw6rz8izCGib+vBjqxAah0pFU06yYGC7BLy56IgKp9J3I5cw6+mRgLBSPFZiCZfS3XPHL7XT2FGwkEK6gbDDg8qqT/tl2ZfGD6N853GJ97MQD2Gpm8S0vzeR3/odCftDvXXNZJxf5tMXtnXtGNcOVBr6rs4KSSGEOkVKeAk8xA62hoz2DDDzActYxXbB6myfa8+q3dXQN0XhgYuw27onJJ5faPSkG4FHiJHuXdUSQKz5OUOsw4AFzeQDYPhQsNyh==

accept;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
4db4276a52606b18093df202d794d9cebaae2e1785ef411ca549520d7a2be21a'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20240312T030211Z
20240312/ap-southeast-2/es/aws4_request
2a099805b1c4f747bdc82cc40cde2826d8e703817ea6a280eeba2c5d7907cf60'
"}
    ... 30 common frames omitted

Is this a bug in the interceptor class?

[dblock]

What was the actual value of the Authorization header sent? Double check it ended up having ap-southeast-2/es/aws4_request and that only the signature was different. Also check what this looks like for a GET.

Stepping back, since the GET request is working, and PUT is not, we can safely assume that the interceptor code is not signing the same body as the one being sent to the server, or appending a header after the signature is calculated. I'd look for what is actually being signed.

Build a sample with your code off https://github.com/dblock/opensearch-java-client-demo/tree/opensearch-2.x (feel free to PR into that project), or make a new project, and we can try to help debug.

[vahidiazar]

Thanks for the help @dblock , I tried to intercept headers for a Get and Put request before and after signing in AwsRequestSigningApacheV5Interceptor. As the next step, I am going to build sample code with the repo you mentioned above.

Headers before signing for a PUT request:

Accept:application/json; charset=UTF-8,
Connection:keep-alive,
Content-Length:47,
Content-Type:application/json; charset=UTF-8,
User-Agent:opensearch-java/2.8.1 (Java/17.0.7),
x-amz-content-sha256:required

Headers after signing for a PUT request:

Accept:application/json; charset=UTF-8,
Authorization:AWS4-HMAC-SHA256 Credential=ASIA5UFMNPHYII5VG2IN/20240313/ap-southeast-2/es/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=309bb0178cd4b780f8bfd2d43a92987dfd0880714f2fc50a8930ca6aa328b68e,
Connection:keep-alive,
Content-Length:47,
Content-Type:application/json; charset=UTF-8,
Host:abc.ap-southeast-2.es.amazonaws.com,
User-Agent:opensearch-java/2.8.1 (Java/17.0.7),
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,
X-Amz-Date:20240313T080742Z,
X-Amz-Security-Token:IQoJb3JpZ2luX2...==

Headers before signing for a GET request:

Accept:application/json; charset=UTF-8,
Connection:keep-alive,
User-Agent:opensearch-java/2.8.1 (Java/17.0.7),
x-amz-content-sha256:required

Headers after signing for a GET request:

Accept:application/json; charset=UTF-8,
Authorization:AWS4-HMAC-SHA256 Credential=ASIA5UFMNPHYII5VG2IN/20240313/ap-southeast-2/es/aws4_request, SignedHeaders=accept;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=585c68ab6d0404437c542e0b96deb0b7628b79d737e6529fecd46de486d2b54f,
Connection:keep-alive,
Host:abc.ap-southeast-2.es.amazonaws.com,
User-Agent:opensearch-java/2.8.1 (Java/17.0.7),
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,
X-Amz-Date:20240313T080741Z,
X-Amz-Security-Token:IQoJb3JpZ2lu...==

[vahidiazar]

I forked the branch and created a sample application (created a PR to the branch you mentioned) accessible here: https://github.com/vahidiazar/opensearch-java-client-demo/tree/opensearch-2.x-ApacheHttpClient5Transport

I could replicate the problem, The app has 2 sections:

1. check cluster health which uses GET request and works fine,
2. Creating an index request which PUT request fails due to signature issue.

Could you please have a look when you free?

[dblock]

The interceptor is seeing a BasicHttpRequest but only knows how to sign a ClassicHttpRequest which has a body (entity). I think this is why we can't implement support for ApacheHttpClient5TransportBuilder in the interceptor, but I could be wrong (and @acm19 or @reta can correct me)?

The ApacheHttpClient5TransportBuilder class is from opensearch-java. There's another dedicated transport option there called AwsSdk2Transport which uses whatever AWS SDK uses and works out of the box. Is there a reason why you can't use that? example on main in my demo.

Another option is to create a client with org.apache.hc.client5.http.impl.async.HttpAsyncClients, see this sample.

[acm19]

As @dblock, what you're experiencing is this issue: https://github.com/acm19/aws-request-signing-apache-interceptor/issues/101#issuecomment-1532174922. I'll re-open it btw, as we should either try to solve it or document that it's a limitation. Back when I checked there didn't seem to be a solution without some hacks.

In your case you're using ApacheHttpClient5TransportBuilder, which is using an async client under the hood that has this problem. If you use a sync client you won't experience the issue. And also there's @dblock recommendation about the out of the box transport, you could give that a try as well and let us know if it helped.

[dblock]

@vahidiazar help us with contributing to the README?

[vahidiazar]

@dblock, Thanks for the response, Initially We were using an old ES client (Jest) which internally using ApacheHttpClient 4. Inorder to hand 429 and internal errors, we were using HttpClientBuilder setRetryHandler and setServiceUnavailableRetryStrategy. After migrating to OpenSearch client, We started to use RestClient first which current release uses ApacheHttpAsyncClient 4 which does not support setRetryHandler and setServiceUnavailableRetryStrategy. That's why I decided to try ApacheHttpClient5Transport.

Regarding your question about why I'm not using AwsSdk2Transport, I have the same problem regarding retrying with it, also not able to disable https on local integration tests.

Any suggestion you have for handling retries?

Happy to help on README, what you expect me to add there?

[dblock]

@vahidiazar It's the first time you mention retries - does the first PUT request actually work, and it's the retry that fails with ApacheHttpClient5Transport? If so that is a different bug. Can you try to put a repro together maybe by modifying my simple demo.

[vahidiazar]

@dblock , Sorry if I confused you. I just tried to clarify why I'm trying to use ApacheHttpClient5Transport. I created a demo here which demonstrates the bug (mentioned above) and it is not related to retry (there is not retry): https://github.com/vahidiazar/opensearch-java-client-demo/tree/opensearch-2.x-ApacheHttpClient5Transport

[dblock]

Thanks @vahidiazar. I'm going to close this issue as a dup of https://github.com/acm19/aws-request-signing-apache-interceptor/issues/101. Thank you for providing an example. If you want to turn it into a failing unit/integration test and PR that, I think we could get one step closer to an implementation, but I am not sure how that could work given the limitations of what the client exposes.