search

awslabs / aws-sdk-kotlin

Multiplatform AWS SDK for Kotlin
Apache License 2.0
396 stars 48 forks source link

IO Exception when listing s3 objects in bucket #1214

Open Sebo11 opened 6 months ago

Sebo11 commented 6 months ago

Describe the bug

When executing below code it randomly happens that the call to s3 fails with an IO Exception (exception is inserted below). I cannot put a precise number on it but I guess this happens once every 20 request or so. It can be "fixed" when the attempts are upped to more than 5. Because when this happens it failes always exactly 5 times with the 6th attempt going trough somehow.

I cannot explain why this occurs and why it always fails 5 times with the 6th attempt going trough.

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.ListObjectsV2Request
import jakarta.inject.Singleton
import io.micronaut.context.annotation.Property

@Singleton
class Service(private val s3: S3Client) : Logger {
    @Property(name = "bucket-name")
    private lateinit var awsbucket: String

    suspend fun listObjects(uuid: UUID): List<Object>? {
        val request = ListObjectsV2Request {
            bucket = awsbucket
            prefix = "documents/$uuid"
        }
        return s3.listObjectsV2(request).contents
    }

with this factory

import aws.sdk.kotlin.services.s3.S3Client
import io.micronaut.context.annotation.Factory
import io.micronaut.context.annotation.Property
import jakarta.inject.Singleton

@Factory
class S3Factory {
    @Singleton
    fun getS3Client(): S3Client = S3Client {
        region = "eu-central-1"
    }
}

the resulting exception 

15:41:51.721 [DefaultDispatcher-worker-17] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
aws.smithy.kotlin.runtime.http.HttpException: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:158)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:209)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:111)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:65)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:205)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:537)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.EOFException: \n not found: limit=0 content=
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335)
        at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179)
        ... 18 common frames omitted

Expected behavior

Not failing randomly exactly 5 times when this error occurs.

Current behavior

Like explained before the error which is thrown. 

15:41:51.721 [DefaultDispatcher-worker-17] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
aws.smithy.kotlin.runtime.http.HttpException: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:158)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:209)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:111)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:65)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:205)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:537)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.EOFException: \n not found: limit=0 content=
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335)
        at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179)
        ... 18 common frames omitted

Here with more attempts set going trough 

09:29:19.565 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(i
d=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.566 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 004111cbb877796271fe307182ce84a0aa2ce6414c9a1e5a74ba3cd038d64e0b
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 2
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 8bc5844c65f86333f4cb21e7206b3ae7e471c8973789db9f86605d02ddc5f0a4
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 3
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 9d2db5c865b79490bb54de1f21d43909e23a0ab9611ebd4f488ad00af09c8f56
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 4
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 74a7dffb53a0ab23be7e32814c42dbe526e96b768e71ba406e13b4a374bb3814
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 5
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 549aae8fd9ed87f1ffdd3117dbfcf69b9522946925a2a97356a31daac8b58aa0
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 6
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: c1e9643a2ffa94e87ad7d1828c7aaf215374963703785314552aefceb0a4c357
09:29:19.832 [DefaultDispatcher-worker-18] DEBUG httpTraceMiddleware - HttpResponse: 200: OK

Steps to Reproduce

The framework in use is Micronaut in the version 4.2.2 . The s3 bucket itself has several objects in it and returns with the prefix key 10 elements.

The code was started locally via Gradle (version 8.4) and invoked over the micronaut controoler via Postman. Note that the same behavior is present in the cloud environment. 

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.ListObjectsV2Request
import jakarta.inject.Singleton
import io.micronaut.context.annotation.Property

@Singleton
class Service(private val s3: S3Client) : Logger {
    @Property(name = "bucket-name")
    private lateinit var awsbucket: String

    suspend fun listObjects(uuid: UUID): List<Object>? {
        val request = ListObjectsV2Request {
            bucket = awsbucket
            prefix = "documents/$uuid"
        }
        return s3.listObjectsV2(request).contents
    }

with this factory

import aws.sdk.kotlin.services.s3.S3Client
import io.micronaut.context.annotation.Factory
import io.micronaut.context.annotation.Property
import jakarta.inject.Singleton

@Factory
class S3Factory {
    @Singleton
    fun getS3Client(): S3Client = S3Client {
        region = "eu-central-1"
    }
}

with these build.gradle.kts file 

plugins {
    id("org.jetbrains.kotlin.jvm") version "1.9.22"
    id("org.jetbrains.kotlin.plugin.allopen") version "1.9.22"
    id("com.google.devtools.ksp") version "1.9.22-1.0.17"
    id("com.github.johnrengelman.shadow") version "8.1.1"
    id("io.micronaut.application") version "4.3.2"
    id("io.micronaut.aot") version "4.3.2"
}

version = "0.1"
group = "com.example"

val kotlinVersion="1.9.22"
repositories {
    mavenCentral()
}

dependencies {
    ksp("io.micronaut:micronaut-http-validation")
    ksp("io.micronaut.serde:micronaut-serde-processor")
    implementation("io.micronaut.kotlin:micronaut-kotlin-runtime")
    implementation("io.micronaut.serde:micronaut-serde-jackson")
    implementation("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:${kotlinVersion}")
    compileOnly("io.micronaut:micronaut-http-client")
    runtimeOnly("ch.qos.logback:logback-classic")
    runtimeOnly("com.fasterxml.jackson.module:jackson-module-kotlin")
    testImplementation("io.micronaut:micronaut-http-client")
    implementation("aws.sdk.kotlin:s3:1.0.54")

}

application {
    mainClass.set("com.example.ApplicationKt")
}
java {
    sourceCompatibility = JavaVersion.toVersion("21")
}

graalvmNative.toolchainDetection.set(false)
micronaut {
    runtime("netty")
    testRuntime("junit5")
    processing {
        incremental(true)
        annotations("com.example.*")
    }
    aot {
    // Please review carefully the optimizations enabled below
    // Check https://micronaut-projects.github.io/micronaut-aot/latest/guide/ for more details
        optimizeServiceLoading.set(false)
        convertYamlToJava.set(false)
        precomputeOperations.set(true)
        cacheEnvironment.set(true)
        optimizeClassLoading.set(true)
        deduceEnvironment.set(true)
        optimizeNetty.set(true)
    }
}

Possible Solution

No response

Context

I want to show the user the s3 objects which the user has saved in the cloud via a web app. However at some point the request fails randomly (locally and in the cloud).

AWS Kotlin SDK version used

1.0.54

Platform (JVM/JS/Native)

Java 21.0.2 with gradle 8.4

Operating System and version

Windows 10 Pro

Sebo11 commented 6 months ago

Hello, I created a new Micronaut project (version 4.3.1) and tested the behavior again. It still fails occasionally with the same error. However, in the fresh application it only needs 2 attempts instead of 6. I can't explain why exactly as the dependencies and the setup are the same in my opinion.

Don't know if it is worth it to look into it.

github-actions[bot] commented 6 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

lauzadis commented 6 months ago

Hi, thanks for the report! I've set up a local application using Micronaut, but I can't replicate this behavior.

Can you please configure logMode = LogMode.LogRequest on the client and share some logs (with sensitive information redacted) demonstrating the problematic requests being retried and then finally succeeding? It will help determine what it changing in the request to make it succeed.

Sebo11 commented 6 months ago

Hello, sorry for the late response. Here the requested logs :)

Like always after the sixth request it somehow gets trough. However, like mentioned before it does not happen a lot. Every 20 request or so. 

15:24:29.029 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(i
d=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.029 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 416437ca1204259b13badc097e54b87aab915e307ddc825c48a9d4efa74b20e3
15:24:29.030 [default-nioEventLoopGroup-1-3] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=1; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=416437ca1204259b13badc097e54b87aab915e307ddc825c48a9d4efa74b20e3       

15:24:29.449 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 2
15:24:29.449 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.450 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 10f692f13983e1b42442bb12f138fbde4ee6f8ea0e1c0fac80bc77b66c6653e1
15:24:29.450 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=2; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=10f692f13983e1b42442bb12f138fbde4ee6f8ea0e1c0fac80bc77b66c6653e1       

15:24:29.795 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 3
15:24:29.796 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.796 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: f3b55ed40eefbb74e0e41ca9496e1029f8ff351d68fdfaa9cea3c37b1ec427b7
15:24:29.797 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=3; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=f3b55ed40eefbb74e0e41ca9496e1029f8ff351d68fdfaa9cea3c37b1ec427b7       

15:24:30.066 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 4
15:24:30.067 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:30.068 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: c46c59d8f3bc4990336f52ebc36fcc8e3b70ce8e037cdf05fcca13aa7ff81937
15:24:30.068 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=4; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142430Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=c46c59d8f3bc4990336f52ebc36fcc8e3b70ce8e037cdf05fcca13aa7ff81937       

15:24:30.315 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
15:24:30.315 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled.
15:24:30.804 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 5
15:24:30.804 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:30.805 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 001bf3ba8b261c022ce2f040ad2c5073cf30d5841e7ee312a89842c28c1d2657
15:24:30.805 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=5; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142430Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=001bf3ba8b261c022ce2f040ad2c5073cf30d5841e7ee312a89842c28c1d2657       

15:24:31.931 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 6
15:24:31.933 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:31.934 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: b443cc2ec0581a82ab2cc410678b2e67eef0a617a718837ac31908d5562f7235
15:24:31.935 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=6; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142431Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=b443cc2ec0581a82ab2cc410678b2e67eef0a617a718837ac31908d5562f7235       

15:24:32.099 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpResponse: 200: OK