Error when putting object in bucket : MD5 does not match etag when using Java AWS SDK

jubos / fake-s3

A lightweight server clone of Amazon S3 that simulates most of the commands supported by S3 with minimal dependencies

2.94k stars 355 forks source link

Error when putting object in bucket : MD5 does not match etag when using Java AWS SDK #164

Open StefH opened 8 years ago

StefH commented 8 years ago

I'm using AWS S3 SDK to put a file into this fake s3.

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to verify integrity of data upload. Client calculated content hash (contentMD5: rtGkDmuy7nJ1ms6ZoEyrKQ== in base 64) didn't match hash (etag: b348ec55910b6836d3f2bd8e69f484f0 in hex) calculated by Amazon S3. You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@78aea4b9, bucketName: hello-world, key: file_636120647747450000.txt) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1609) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3147) at xxx.aws.s3.S3Context.putObjectAsString(S3Context.java:26)

l15k4 commented 7 years ago

I think it's this https://github.com/jubos/fake-s3/issues/107

l15k4 commented 7 years ago

I investigated further and the last version of aws-java-sdk-s3 that fake-s3 works with is 1.11.15 ... A breaking change must have been introduced in 1.11.16

cescude commented 7 years ago

FWIW, it looks like the Amazon client uses chunked encoding by default now (rather than sending the raw bytes), and fake-s3 ends up calculating the md5 on encoded payload, not the raw data.

You can check by navigating to where fakes3 stores the data, running md5sum content, and comparing that to the md5sum of the original file.

A quick workaround is to initialize the AmazonS3Client with chunked-mode disabled, i.e.:

s3client.setS3ClientOptions(
  S3ClientOptions.builder
    .setPathStyleAccess(true)
    .disableChunkedEncoding.build)

When I do this locally, everything works fine.

(I'm running with the AWS SDK version 1.11.41)

kknd22 commented 6 years ago

can we have a cleaner fix other than this flag?

kknd22 commented 6 years ago

this make the production code look like:

if (USING_FASKES3)
  AmazonS3ClientBuilder.standard().disableChunkedEncoding
else 
    AmazonS3ClientBuilder.standard()

should have better solution?

-cl

jubos commented 6 years ago

I agree that this is a pretty clunky solution. Will investigate a solution and see if it is easy to get on the roadmap.

jensmith-work commented 6 years ago

the work-around works but as kknd22 commented, I would also like a cleaner solution