jubos / fake-s3

A lightweight server clone of Amazon S3 that simulates most of the commands supported by S3 with minimal dependencies
2.94k stars 355 forks source link

aws java sdk wants uppercase headers (ETag) #107

Open bytesandwich opened 9 years ago

bytesandwich commented 9 years ago

per https://github.com/aws/aws-sdk-java/issues/436

https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/Headers.java#L34 and https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/ObjectMetadata.java#L625

the aws java sdk expects headers to be in a particular case (not lowercase), even though headers are supposed to be case insensitive, and webrick makes all header keys lowercase.

I can't see how to get webrick to not lowercase all headers, and I'm unfamiliar with ruby generally, so I'm going to add a proxy for my own use that sets the case but it's pretty lame.

tom-leahy commented 9 years ago

Here's what I see of the S3 code doing the web response parsing to S3Object (and metadata): https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/AbstractS3ResponseHandler.java (protected void populateObjectMetadata(HttpResponse response, ObjectMetadata metadata)) And what is in the Headers.S3_USER_METADATA_PREFIX variable (https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/Headers.java): "x-amz-meta-"

I threw a breakpoint in and I'm seeing headers come back from fake-s3 in the format of "X-Amz-Meta-Foobar" - and this caps doesn't match AWS's startsWith check using the lowercase value and that's why I don't see any user metadata come through. Looking at the server.rb it looks like this header is being written as lowercase, is it possible that WEBrick is somehow doing something to the headers on the way out?

declension commented 8 years ago

I'm seeing similar behaviour with the Java client - metadata.getRawMetadata() gives:

{Accept-Ranges=bytes, Last-Ranges=bytes, ETag=a112192ed08ced89a3960ff183b24f53, Connection=Keep-Alive, Last-Modified=Wed Mar 23 11:21:22 GMT 2016, Content-Length=87927, X-Amz-Meta-User-Id=448f0ce0-f0e9-11e5-a8b9-c31e3e141217, Content-Type=application/octet-stream}

whereas metadata.getUserMetadataOf("user-id") returns null, as does using "User-Id".

pickhardt commented 8 years ago

Is this really a bug with Fake S3, or is it a bug with the Java SDK?

gaul commented 8 years ago

Newer versions of the Java SDK fix this: aws/aws-sdk-java#590

l15k4 commented 7 years ago

Imho it doesn't, I still get the same error using :

com.amazonaws.services.s3.AmazonS3Client#putObject(java.lang.String, java.lang.String, java.io.File)

and aws-java-sdk-s3 1.11.46