Closed turboDi closed 1 month ago
Hi @turboDi, thanks for the bug report! Unfortunately, I cannot reproduce this locally on SDK version 1.2.50. I've uploaded several objects to S3 with no contentEncoding
parameter specified and they appear in S3 without that metadata included.
Can you please enable request logging and post the trace log of the API call with any sensitive information redacted?
Example of enabling request logging:
S3Client {
region = "us-west-2"
logMode = LogMode.LogRequest // Log request headers but not the body
}
Hi @ianbotsf, thanks for a quick response! Here is a trace log for the test
2024-07-12 19:31:13.929 TRACE [main @coroutine#1] a.s.k.r.h.o.OperationHandler operation started ||
2024-07-12 19:31:14.027 TRACE [main @coroutine#1] a.s.k.r.a.a.CachedCredentialsProvider refreshing credentials cache ||
2024-07-12 19:31:14.029 TRACE [main @coroutine#1] a.s.k.r.i.IdentityProviderChain Attempting to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.SystemPropertyCredentialsProvider@3b152928 ||
2024-07-12 19:31:14.031 TRACE [main @coroutine#1] a.s.k.r.a.c.SystemPropertyCredentialsProvider Attempting to load credentials from system properties aws.accessKeyId/aws.secretAccessKey/aws.sessionToken ||
2024-07-12 19:31:14.032 DEBUG [main @coroutine#1] a.s.k.r.i.IdentityProviderChain unable to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.SystemPropertyCredentialsProvider@3b152928: Missing value for system property `aws.accessKeyId` ||
2024-07-12 19:31:14.034 TRACE [main @coroutine#1] a.s.k.r.i.IdentityProviderChain Attempting to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.EnvironmentCredentialsProvider@a0a9fa5 ||
2024-07-12 19:31:14.035 TRACE [main @coroutine#1] a.s.k.r.a.c.EnvironmentCredentialsProvider Attempting to load credentials from env vars AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN ||
2024-07-12 19:31:14.040 DEBUG [main @coroutine#1] a.s.k.r.h.o.AuthHandler resolved endpoint: Endpoint(uri=https://test-bucket.s3.us-east-2.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-2, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]}) ||
2024-07-12 19:31:14.042 DEBUG [main @coroutine#1] a.s.k.r.h.i.FlexibleChecksumsRequestInterceptor no checksum algorithm specified, skipping flexible checksums processing ||
2024-07-12 19:31:14.058 TRACE [main @coroutine#1] a.s.k.r.a.a.DefaultAwsSignerImpl Canonical request:
PUT
/test.zip
x-id=PutObject
amz-sdk-invocation-id:d6be1597-6d5d-4c4b-9fad-457dc050acfe
amz-sdk-request:attempt=1; max=3
content-encoding:aws-chunked
content-type:application/zip
host:test-bucket.s3.us-east-2.amazonaws.com
transfer-encoding:chunked
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20240712T153114Z
x-amz-decoded-content-length:1307372
x-amz-user-agent:aws-sdk-kotlin/1.2.50
amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-user-agent
STREAMING-AWS4-HMAC-SHA256-PAYLOAD ||
2024-07-12 19:31:14.060 TRACE [main @coroutine#1] a.s.k.r.a.a.DefaultAwsSignerImpl String to sign:
AWS4-HMAC-SHA256
20240712T153114Z
20240712/us-east-2/s3/aws4_request
22330e7b0542e58dd369be91282b100edaf70bb3748ae982b4b54c3a9125bfd5 ||
2024-07-12 19:31:14.060 DEBUG [main @coroutine#1] a.s.k.r.a.a.DefaultAwsSignerImpl Calculated signature: <REDACTED> ||
2024-07-12 19:31:14.077 DEBUG [main @coroutine#1] httpTraceMiddleware HttpRequest:
PUT /test.zip?x-id=PutObject
Host: test-bucket.s3.us-east-2.amazonaws.com
Content-Type: application/zip
User-Agent: aws-sdk-kotlin/1.2.50 ua/2.1 api/s3#1.2.50 os/macos#13.6.7 lang/kotlin#1.9.23 md/javaVersion#17.0.9 md/jvmName#OpenJDK_64-Bit_Server_VM md/jvmVersion#17.0.9+0 m/E
x-amz-user-agent: aws-sdk-kotlin/1.2.50
amz-sdk-invocation-id: d6be1597-6d5d-4c4b-9fad-457dc050acfe
amz-sdk-request: attempt=1; max=3
Content-Encoding: aws-chunked
Transfer-Encoding: chunked
X-Amz-Decoded-Content-Length: 1307372
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20240712T153114Z
Authorization: AWS4-HMAC-SHA256 Credential=<REDACTED>/20240712/us-east-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-user-agent, Signature=<REDACTED>
||
2024-07-12 19:31:14.107 TRACE [DefaultDispatcher-worker-1 @call-context#2] a.s.k.r.h.e.o.OkHttpEngine call started ||
2024-07-12 19:31:14.119 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine proxy select start: url=https://test-bucket.s3.us-east-2.amazonaws.com/ ||
2024-07-12 19:31:14.122 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine proxy select end: url=https://test-bucket.s3.us-east-2.amazonaws.com/; proxies=[DIRECT] ||
2024-07-12 19:31:14.122 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine dns query: domain=test-bucket.s3.us-east-2.amazonaws.com ||
2024-07-12 19:31:14.200 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine dns resolved: domain=test-bucket.s3.us-east-2.amazonaws.com; records=[test-bucket.s3.us-east-2.amazonaws.com/52.219.179.42, test-bucket.s3.us-east-2.amazonaws.com/3.5.128.171, test-bucket.s3.us-east-2.amazonaws.com/3.5.130.118, test-bucket.s3.us-east-2.amazonaws.com/3.5.132.120, test-bucket.s3.us-east-2.amazonaws.com/3.5.131.123, test-bucket.s3.us-east-2.amazonaws.com/3.5.128.100, test-bucket.s3.us-east-2.amazonaws.com/3.5.129.118, test-bucket.s3.us-east-2.amazonaws.com/52.219.92.194] ||
2024-07-12 19:31:14.206 TRACE [OkHttp connect https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine starting connection: addr=test-bucket.s3.us-east-2.amazonaws.com/52.219.179.42:443; proxy=DIRECT ||
2024-07-12 19:31:14.408 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine initiating TLS connection ||
2024-07-12 19:31:14.762 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine TLS connect end: handshake=Handshake{tlsVersion=TLS_1_3 cipherSuite=TLS_AES_128_GCM_SHA256 peerCertificates=[CN=*.s3.us-east-2.amazonaws.com, CN=Amazon RSA 2048 M01, O=Amazon, C=US, CN=Amazon Root CA 1, O=Amazon, C=US] localCertificates=[]} ||
2024-07-12 19:31:14.765 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine connection established: addr=test-bucket.s3.us-east-2.amazonaws.com/52.219.179.42:443; proxy=DIRECT; protocol=http/1.1 ||
2024-07-12 19:31:14.767 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine connection acquired: conn(id=1330540524)=Connection{test-bucket.s3.us-east-2.amazonaws.com:443, proxy=DIRECT hostAddress=test-bucket.s3.us-east-2.amazonaws.com/52.219.179.42:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http/1.1}; connPool: total=1, idle=0 ||
2024-07-12 19:31:14.770 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine sending request headers ||
2024-07-12 19:31:14.771 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine finished sending request headers ||
2024-07-12 19:31:14.772 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine sending request body ||
2024-07-12 19:31:15.947 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine finished sending request body: bytesSent=1309257 ||
2024-07-12 19:31:16.333 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine response headers start ||
2024-07-12 19:31:16.334 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine response headers end: contentLengthHeader=0 ||
2024-07-12 19:31:16.336 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine response body available ||
2024-07-12 19:31:16.336 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine response body finished: bytesConsumed=0 ||
2024-07-12 19:31:16.337 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine connection released: conn(id=1330540524)=Connection{test-bucket.s3.us-east-2.amazonaws.com:443, proxy=DIRECT hostAddress=test-bucket.s3.us-east-2.amazonaws.com/52.219.179.42:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http/1.1}; connPool: total=1, idle=1 ||
2024-07-12 19:31:16.338 TRACE [OkHttp https://test-bucket.s3.us-east-2.amazonaws.com/...] a.s.k.r.h.e.o.OkHttpEngine call complete ||
2024-07-12 19:31:16.344 DEBUG [main @coroutine#1] httpTraceMiddleware HttpResponse: 200: OK ||
2024-07-12 19:31:16.366 TRACE [main @coroutine#1] a.s.k.r.h.o.OperationHandler operation completed successfully ||
Ah, I see why I wasn't able to reproduce this: I was using too small of a test object!
Since you're uploading an object over 1MB, the SDK is automatically using chunked encoding. This feature allows streaming the payload to S3 without first reading it to calculate the request signature.
Unfortunately, it appears using chunked encoding has a side effect. From the page linked above:
If you specify Content-Encoding in your request as
Content-Encoding : aws-chunked
, S3 adds an empty value forContent-Encoding
and stores the object metadata (Content-Encoding :
) to the resulting object.
One way to get around this is to disable the use of chunked encoding on your client:
S3Client {
region = "us-west-2"
enableAwsChunked = false
}
What kind of errors/problems are you seeing when downloading S3 objects with Content-Encoding :
?
I see, thank you for detailed answer, enableAwsChunked
did the trick.
I'm using ktor
http client to download s3 objects with presigned url. This presigned url returns Content-Encoding :
header and ktor
doesn't support this out of the box.
I can tweak my ktor client to support this empty encoding, but that looked like a bug to me and made me think that our other clients might be confused with this non-standard header value.
Anyway, this is not an sdk issue and I think we can close it
I found a cross-SDK issue about this: https://github.com/aws/aws-sdk/issues/498. I'm checking internally to see what the status with S3 is but there may actually be a bug here which could be fixed by S3 or the SDKs.
I've confirmed that this is indeed an S3 bug and a fix is being worked on which would address the issue independent of the SDK. I don't have an estimated timeline to share so, in the meantime, you'll have to either:
Content-Encoding
header from GetObject callsPlease track https://github.com/aws/aws-sdk/issues/498 for updates on the S3 fix.
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Describe the bug
S3Client is uploading files with empty
Content-Encoding
metadata ifcontentEncoding
parameter is not specifiedExpected behavior
If
contentEncoding
is not specified, S3Client shouldn't add emptyContent-Encoding
metadata. I expect the same result as this aws cli call:aws s3 cp test.zip s3://test-bucket/test.zip
Current behavior
If
contentEncoding
is not specified, S3Client adds emptyContent-Encoding
metadataSteps to Reproduce
Possible Solution
No response
Context
When files with empty
Content-Encoding
are downloaded back from s3 with direct links or other clients like ktor, they fail on emptyContent-Encoding
header. Have to add hacks to treat emptyContent-Encoding
as identity. But for clients, that I don't have control over, there is no workaround.AWS SDK for Kotlin version
1.2.50
Platform (JVM/JS/Native)
JVM
Operating system and version
MacOS, Amazon Linux 2