search

vyadh / teamcity-azure-keyvault-plugin

Azure Key Vault TeamCity Plugin
Apache License 2.0
8 stars 2 forks source link

Improve resiliency in the face of Azure Key Vault timeouts #3

Closed vyadh closed 5 years ago

vyadh commented 5 years ago

Azure Key Vault regularly times out in response to requests, particularly from midday UK time (America waking up?).

Microsoft does provide some advice although this appears to be if your own services are requesting too often and it returns a 4xx code, which is not what I'm seeing.

For example:

Error processing parameters for Azure Key Vault: Read timed out
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at okio.Okio$2.read(Okio.java:140)
    at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
    at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358)
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
    at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at com.github.vyadh.teamcity.keyvault.agent.AzureKeyVaultConnector.executeRequest(AzureKeyVaultConnector.kt:36)
    at com.github.vyadh.teamcity.keyvault.agent.AzureKeyVaultConnector.requestValue(AzureKeyVaultConnector.kt:28)
    at com.github.vyadh.teamcity.keyvault.agent.KeyVaultBuildFeature$fetchSecrets$2.apply(KeyVaultBuildFeature.kt:88)
    at com.github.vyadh.teamcity.keyvault.agent.KeyVaultBuildFeature$fetchSecrets$2.apply(KeyVaultBuildFeature.kt:17)
    at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
    at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:390)
    at java.util.stream.DistinctOps$1$2.end(DistinctOps.java:168)
    at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
    at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.github.vyadh.teamcity.keyvault.agent.KeyVaultBuildFeature.fetchSecrets(KeyVaultBuildFeature.kt:86)
    at com.github.vyadh.teamcity.keyvault.agent.KeyVaultBuildFeature.buildStarted(KeyVaultBuildFeature.kt:36)
    at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:128)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
    at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:122)
    at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:72)
    at com.sun.proxy.$Proxy8.buildStarted(Unknown Source)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireBuildStartedStage.doBuildState(FireBuildStartedStage.java:20)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireEventStageBase.doBuildStage(FireEventStageBase.java:26)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:75)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:55)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:300)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:54)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:264)
    at java.lang.Thread.run(Thread.java:748)
vyadh commented 5 years ago

Turns out that the default connect and read timeouts in OkHttp is 10s.

For now, I'll set each to 30s to see if that improves things.

There is a risk that it will timeout regardless at that it's hitting the 10s limit reflects it's never going to work. We'll have to try it in practice to be sure, but minor updates can update the value, learning through practical experience.