Open rcohenDBRS opened 5 years ago
Please find corresponding lines in agent's teamcity-agent.log log file and post them here. There would be some stacktrace with detailed error message.
Thanks for the quick response.
[2019-01-29 16:19:51,389] WARN - .agent.VaultParametersResolver - Failed to fetch data for path '/de/prod/model-deployer'
org.springframework.vault.VaultException: Status 403 de/prod/model-deployer: permission denied; nested exception is org.springframework.web.client.HttpClientErrorException: 403 Forbidden
at org.jetbrains.teamcity.vault.support.VaultResponses.buildException(VaultResponses.java:82)
at org.jetbrains.teamcity.vault.support.VaultTemplate$3.doWithRestOperations(VaultTemplate.java:180)
at org.jetbrains.teamcity.vault.support.VaultTemplate.doWithSession(VaultTemplate.java:159)
at org.jetbrains.teamcity.vault.support.VaultTemplate.doRead(VaultTemplate.java:167)
at org.jetbrains.teamcity.vault.support.VaultTemplate.read(VaultTemplate.java:128)
at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.fetch(VaultParametersResolver.kt:94)
at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.doFetchAndPrepareReplacements(VaultParametersResolver.kt:82)
at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:74)
at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:70)
at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.resolve(VaultParametersResolver.kt:51)
at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:136)
at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:33)
at org.jetbrains.teamcity.vault.UtilKt.activity(util.kt:155)
at org.jetbrains.teamcity.vault.agent.VaultBuildFeature.buildStarted(VaultBuildFeature.kt:79)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:126)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:120)
at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:70)
at com.sun.proxy.$Proxy7.buildStarted(Unknown Source)
at jetbrains.buildServer.agent.impl.buildStages.startStages.FireBuildStartedStage.doBuildState(FireBuildStartedStage.java:20)
at jetbrains.buildServer.agent.impl.buildStages.startStages.FireEventStageBase.doBuildStage(FireEventStageBase.java:26)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:75)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:55)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:300)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:54)
at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:264)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.HttpClientErrorException: 403 Forbidden
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:264)
at org.jetbrains.teamcity.vault.support.VaultTemplate$3.doWithRestOperations(VaultTemplate.java:173)
... 33 more
[2019-01-29 16:19:51,389] WARN - .agent.VaultParametersResolver - Cannot resolve '/de/prod/model-deployer!/model-deployer-access-key': data wasn't received from HashiCorp Vault
[2019-01-29 16:19:51,390] WARN - .agent.VaultParametersResolver - Cannot resolve '/de/prod/model-deployer!/model-deployer-secret-key': data wasn't received from HashiCorp Vault
Also, I'm not sure if this is related, but in teamcity-server.log, with the default policy applied for the approle, I get this error:
[2019-01-29 16:11:42,078] WARN - ty.vault.server.VaultConnector - Failed to revoke token: org.springframework.web.client.HttpClientErrorException: 403 Forbidden (enable debug to see stacktrace)
When I added
path "auth/token/revoke-accessor" {
capabilities = ["read", "list", "create", "update"]
}
to the policy, the error changed to
[2019-01-29 16:19:52,302] WARN - ty.vault.server.VaultConnector - Failed to revoke token: org.springframework.web.client.HttpClientErrorException: 400 Bad Request (enable debug to see stacktrace)
For anyone with a similar problem, a workaround I found was to create a new vault connection with a different namespace for every secret I wanted to access.
@VladRassokhin Any answers?
For anyone with a similar problem, a workaround I found was to create a new vault connection with a different namespace for every secret I wanted to access.
Hi @rcohenDBRS, I have same issue.
I am trying to read one secret for test purpose but the property within the build script appears as a single string like %password%
. Parameter namespace not empty in this scenario.
Without parameter namespace I get 403 error message within client log plus Failed to revoke token within server log.
@VladRassokhin any idea for this issue due to looks like it is not an unique problem.
Edit I have managed it and it works now. Would be really beneficial if someone could update the documentation!
@realrill What did you manage?
@rcohenDBRS Reading (multiple) secrets, but only with your solution. I have had incorrect parameter, that was the source of my issue.
same here. Can someone please explain the workaround in more detailed ways?
When, in one project, I try to set environment variables with kvs from multiple secrets, I get this error:
If I only try to get from one secret, there are no issues.