SocketTimeoutException during SSL handshake when using StsWebIdentityCredentialsProvider in AWS Kotlin SDK

[vineetu]

Describe the bug

I’m encountering a SocketTimeoutException when using the AWS Kotlin SDK to connect to DynamoDB in a Kubernetes environment. The exception occurs during the SSL handshake when the StsWebIdentityCredentialsProvider attempts to retrieve credentials.

When I do the same using the java alternative software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider. It works just fine.

Here is the stack

Exception in thread "main" aws.smithy.kotlin.runtime.http.HttpException: java.net.SocketTimeoutException: Read timed out; HttpErrorCode(SOCKET_TIMEOUT) at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:168) at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:99) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:832) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379) at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:32) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected behavior

The AWS Kotlin SDK should successfully use the StsWebIdentityCredentialsProvider to obtain temporary credentials and communicate with AWS services without encountering a socket timeout.

Current behavior

There is a socket timeout and the application throws a SocketTimeoutException during the SSL handshake when attempting to retrieve credentials using the StsWebIdentityCredentialsProvider.

Steps to Reproduce

dynamoDB = DynamoDbClient { region = dynamoConfig.region credentialsProvider = StsWebIdentityCredentialsProvider.fromEnvironment(roleArn = "arn", roleSessionName = "name", webIdentityTokenFilePath= "token-location", region = "us-west-2") }

Possible Solution

No response

Context

I am no longer able to call dynamo-db. I am moving my service from java to kotlin and wanted to use the kotlin sdk. Now this is blocker.

AWS SDK for Kotlin version

1.3.57

Platform (JVM/JS/Native)

JVM

Operating system and version

runs in an kubernetes pod in an ec2 container

[ianbotsf]

Hi @vineetu, I'm sorry you're seeing socket timeout exceptions connecting to STS. I'm unable to reproduce this on a regular (i.e., non-Kubernetes) EC2 instance. I'm working on setting up EKS now but, in the meantime, can you please provide some additional info for debugging:

• Do any AWS SDK calls (using non-STS credentials) succeed in your environment? That is, can you use local credentials to access DynamoDB directly?
• Does your environment use a VPC? Is that VPC configured to allow secure access to STS endpoints?
• Can you share the Java AWS SDK code which succeeds?