Spring Cloud Vault Config - Secretes Backend AWS - SocketTimeoutException

[randyhbh]

Describe the bug I have a Spring Boot + Kotlin microservice that gets the AWS credentials to connect to DynamoDB from Vault.

But randomly the Vault implementation is throwing a SocketTimeoutException while renewing the session token, see the stack trace

org.springframework.web.client.ResourceAccessException: I/O error on GET request for \"https://my.vault.server:443/v1/aws/sts/prod-documentrecognitionservice\": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711)
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:334)
    at org.springframework.vault.core.VaultTemplate.lambda$doRead$5(VaultTemplate.java:461)
    at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:448)
    at org.springframework.vault.core.VaultTemplate.doRead(VaultTemplate.java:458)
    at org.springframework.vault.core.VaultTemplate.read(VaultTemplate.java:353)
    at org.springframework.vault.core.lease.SecretLeaseContainer.doGetSecrets(SecretLeaseContainer.java:645)
    at org.springframework.vault.core.lease.SecretLeaseContainer.doStart(SecretLeaseContainer.java:390)
    at org.springframework.vault.core.lease.SecretLeaseContainer.onLeaseExpired(SecretLeaseContainer.java:742)
    at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRotation$3(SecretLeaseContainer.java:595)
    at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:862)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1455)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1059)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:109)
    at org.springframework.vault.core.VaultTemplate.lambda$getSessionInterceptor$1(VaultTemplate.java:255)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
    at org.springframework.vault.client.RestTemplateBuilder.lambda$createTemplate$4(RestTemplateBuilder.java:239)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
    at org.springframework.vault.client.VaultClients.lambda$createRestTemplate$0(VaultClients.java:122)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
    at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:77)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
    ... 19 common frames omitted

In the documentation, how I can provide my own RestTemplate configuration increasing the timeout and retrying is unclear.

Is there a way I can provide my own RestTemplate configuration to Vault?

I did the question on StackOverflow but without an answer there, so I decided to open one here to try to get help.

[randyhbh]

Please @mp911de could you give me some guidance here on how to achieve this 🙏

[spectateur]

Hey @randyhbh , have you been able to collect tcpdump trace of this event ? We are also experiencing random timeout while renewing token / authenticating with the vault. We didn't catch the event on the network level yet and expect to do so to share our findings...

[randyhbh]

Hi @spectateur, unfortunately not, I was checking our monitoring that we have in Vault and I could not spot anything of the like.

That is why I was looking into increasing the timeout or retrying on this issue.

Did you find a solution for this @spectateur?

[mp911de]

This is a duplicate of #659

[randyhbh]

Hi, @mp911de thanks for linking the previous issue, indeed is duplicated, but I could not find in the docs a way to provide to spring cloud vault with my own Apache HTTP Client config, how can I achieve that? Could you point me in the right direction?

[mp911de]

You can fully customize the client, through depending on the level of the customization your mileage may vary. The simple-most option is to set the timeout via ClientOptions.

A more elaborate variant could look like the following code:

HttpClientBuilder builder = HttpClientBuilder.create();

// customize builder

HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(builder.build());

RestTemplateBuilder templateBuilder = RestTemplateBuilder.builder()
        .requestFactory(requestFactory)
        .endpoint(VaultEndpoint.create("vault.com", 8200));

SessionManager sessionManager = // setup authentication

VaultTemplate template = new VaultTemplate(templateBuilder, sessionManager);

[spectateur]

@randyhbh I created an issue on Apache Jira https://issues.apache.org/jira/browse/HTTPCLIENT-2235

[spectateur]

https://github.com/spring-cloud/spring-cloud-vault/issues/660