search

JetBrains / teamcity-hashicorp-vault-plugin

TeamCity plugin to support HashiCorp Vault
Apache License 2.0
28 stars 18 forks source link

Retry request to Vault if it fails because of network connection #48

Closed yaegor closed 1 year ago

yaegor commented 3 years ago

If request to Vault fails, it makes sense to retry before failing the build. This is probbaly actual for the server and agent operations.

BTW, in case of the network errors, it would make sense to include the original error in the build log message.

Here is an example of the failing agent operation: Build log:

Failed to fetch data for path '/aws-XXX/sts/XXX'
Cannot resolve '/aws-XXX/sts/XXX!/access_key': data wasn't received from HashiCorp Vault

Agent log:

[2021-11-22 14:34:29,052]   WARN - .agent.VaultParametersResolver - Failed to fetch data for path '/aws-XXX/sts/XXX'
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://vault.XXX:443/v1/aws-XXX/sts/XXX": Connect to vault.XXX:443 [...] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to vault.XXX:443 [...] failed: connect timed out
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:607)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:264)
    at org.jetbrains.teamcity.vault.support.VaultTemplate$3.doWithRestOperations(VaultTemplate.java:180)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.doWithSession(VaultTemplate.java:166)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.doRead(VaultTemplate.java:174)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.read(VaultTemplate.java:135)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.fetch(VaultParametersResolver.kt:94)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.doFetchAndPrepareReplacements(VaultParametersResolver.kt:82)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:74)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:70)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.resolve(VaultParametersResolver.kt:51)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:138)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:35)
    at org.jetbrains.teamcity.vault.UtilKt.activity(util.kt:171)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature.buildStarted(VaultBuildFeature.kt:79)
    at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:138)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
    at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:132)
    at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:82)
    at com.sun.proxy.$Proxy7.buildStarted(Unknown Source)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireBuildStartedStage.doBuildState(FireBuildStartedStage.java:20)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireEventStageBase.doBuildStage(FireEventStageBase.java:28)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:33)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:76)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:35)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:81)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:57)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:332)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$200(BuildAgentImpl.java:58)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:287)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to vault.XXX:443 [...] failed: connect timed out
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:134)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:319)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:91)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
    at org.jetbrains.teamcity.vault.support.VaultTemplate$1.intercept(VaultTemplate.java:104)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:85)
    at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:69)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:596)
    ... 36 more
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:244)
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:125)
    ... 55 more
andrecp commented 1 year ago

This would be helpful, we see this with some frequency:

"[https://prod.vault.com:443/v1/auth/approle/teamcity/root/login":](https://prod.vault.com/v1/auth/approle/teamcity/root/login%22:) Connect to prod.vault.com:443 [prod.vault.com/, prod.vault.com/, prod.vault.com/] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to prod.vault.com:443 [prod.vault.com/, prod.vault.com2, prod.vault.com7] failed: connect timed out, see teamcity-server.log for details
SocksDevil commented 1 year ago

Thanks for the reminder on this! We've created a task for it on Youtrack: TW-81838