search

findify / s3mock

Embedded S3 server for easy mocking
MIT License
386 stars 107 forks source link

S3AsyncClient issue #147

Open AceHack opened 5 years ago

AceHack commented 5 years ago

When trying to perform any action using the S3AsyncClient I get the following error. Works fine with real S3 server also, works fine when using S3Client and s3mock.

[WARN] [06/07/2019 17:02:57.707] [s3mock-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(s3mock)] Illegal request, responding with status '400 Bad Request': Request is missing required Host header

AceHack commented 5 years ago

I should mention I'm using the AnonymousCredentialsProvider.

sworisbreathing commented 4 years ago

@AceHack I get this issue as well. This was logged previously as https://github.com/aws/aws-sdk-java-v2/issues/429 however I'm seeing it even with version 2.9.15 of the aws sdk.

Stepping through the code, it appears AWS's Apache httpclient implementation implicitly sets the host header, but the netty implementation doesn't and neither does the native java implementation

sworisbreathing commented 4 years ago

Update: I'm still digging into the AWS SDK code to find out where it's forgetting to set the host header. From the tests I'm running it seems to affect the S3AsyncClient but not DynamoDbAsyncClient or SqsAsyncClient, so I'm not entirely sure the problem is in the Netty Implementation (which is linked above this comment).

In the meantime, here's a workaround you can do in your tests:

Map<String, List<String>> S3_HOST_HEADERS = new HashMap<>();
S3_HOST_HEADERS.put("Host", Collections.singletonList("localhost:" + S3_PORT));

S3AsyncClient client = S3AsyncClient.builder()
  // ... other configs here
  .overrideConfiguration(builder -> builder.headers(S3_HOST_HEADERS))
  .serviceConfiguration(builder -> builder.pathStyleAccessEnabled(Boolean.TRUE))
  .build();
sworisbreathing commented 4 years ago

Update: found the issue, reported as https://github.com/aws/aws-sdk-java-v2/issues/1473

Alternative workaround to the above is to use something other than anonymous credentials when setting up the S3AsyncClient. This will ensure that the underlying HTTP requests are properly signed, which will implicitly set the Host header. S3Mock doesn't seem to care what you provide, credentials-wise, so ideally you would just use:

S3AsyncClient client = S3AsyncClient.builder()
    .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("x", "x")))
    .build();

However due to #10 this doesn't seem to work very well. So in the end I'd still stick with anonymous authentication and explicitly overriding the host header