Cannot get kvs from multiple secrets

[rcohenDBRS]

When, in one project, I try to set environment variables with kvs from multiple secrets, I get this error:

[HashiCorp Vault ('de_stage' namespace)] 4 Vault references to resolve: [vault:de_stage:/de/stage/external-model-ui!/external-model-ui-qa-access-key, vault:de_stage:/de/stage/external-model-ui!/external-model-ui-qa-secret-key, vault:de_stage:/de/stage/model-deployer!/model-deployer-access-key, vault:de_stage:/de/stage/model-deployer!/model-deployer-secret-key]
[15:17:13]
[HashiCorp Vault ('de_stage' namespace)] Failed to fetch data for path '/de/stage/model-deployer'
[15:17:13]
[HashiCorp Vault ('de_stage' namespace)] Cannot resolve '/de/stage/model-deployer!/model-deployer-access-key': data wasn't received from HashiCorp Vault
[15:17:13]
[HashiCorp Vault ('de_stage' namespace)] Cannot resolve '/de/stage/model-deployer!/model-deployer-secret-key': data wasn't received from HashiCorp Vault

If I only try to get from one secret, there are no issues.

[VladRassokhin]

Please find corresponding lines in agent's teamcity-agent.log log file and post them here. There would be some stacktrace with detailed error message.

[rcohenDBRS]

Thanks for the quick response.

[2019-01-29 16:19:51,389]   WARN - .agent.VaultParametersResolver - Failed to fetch data for path '/de/prod/model-deployer'
org.springframework.vault.VaultException: Status 403 de/prod/model-deployer: permission denied; nested exception is org.springframework.web.client.HttpClientErrorException: 403 Forbidden
    at org.jetbrains.teamcity.vault.support.VaultResponses.buildException(VaultResponses.java:82)
    at org.jetbrains.teamcity.vault.support.VaultTemplate$3.doWithRestOperations(VaultTemplate.java:180)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.doWithSession(VaultTemplate.java:159)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.doRead(VaultTemplate.java:167)
    at org.jetbrains.teamcity.vault.support.VaultTemplate.read(VaultTemplate.java:128)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.fetch(VaultParametersResolver.kt:94)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver$VaultParametersFetcher.doFetchAndPrepareReplacements(VaultParametersResolver.kt:82)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:74)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.doFetchAndPrepareReplacements(VaultParametersResolver.kt:70)
    at org.jetbrains.teamcity.vault.agent.VaultParametersResolver.resolve(VaultParametersResolver.kt:51)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:136)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature$buildStarted$$inlined$forEach$lambda$1.invoke(VaultBuildFeature.kt:33)
    at org.jetbrains.teamcity.vault.UtilKt.activity(util.kt:155)
    at org.jetbrains.teamcity.vault.agent.VaultBuildFeature.buildStarted(VaultBuildFeature.kt:79)
    at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:126)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
    at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:120)
    at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:70)
    at com.sun.proxy.$Proxy7.buildStarted(Unknown Source)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireBuildStartedStage.doBuildState(FireBuildStartedStage.java:20)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.FireEventStageBase.doBuildStage(FireEventStageBase.java:26)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:75)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:55)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:300)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:54)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:264)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.HttpClientErrorException: 403 Forbidden
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:264)
    at org.jetbrains.teamcity.vault.support.VaultTemplate$3.doWithRestOperations(VaultTemplate.java:173)
    ... 33 more
[2019-01-29 16:19:51,389]   WARN - .agent.VaultParametersResolver - Cannot resolve '/de/prod/model-deployer!/model-deployer-access-key': data wasn't received from HashiCorp Vault
[2019-01-29 16:19:51,390]   WARN - .agent.VaultParametersResolver - Cannot resolve '/de/prod/model-deployer!/model-deployer-secret-key': data wasn't received from HashiCorp Vault

[rcohenDBRS]

Also, I'm not sure if this is related, but in teamcity-server.log, with the default policy applied for the approle, I get this error:

[2019-01-29 16:11:42,078]   WARN - ty.vault.server.VaultConnector - Failed to revoke token: org.springframework.web.client.HttpClientErrorException: 403 Forbidden (enable debug to see stacktrace)

When I added

path "auth/token/revoke-accessor" {
  capabilities = ["read", "list", "create", "update"]
}

to the policy, the error changed to

[2019-01-29 16:19:52,302]   WARN - ty.vault.server.VaultConnector - Failed to revoke token: org.springframework.web.client.HttpClientErrorException: 400 Bad Request (enable debug to see stacktrace)

[rcohenDBRS]

For anyone with a similar problem, a workaround I found was to create a new vault connection with a different namespace for every secret I wanted to access.

[rcohenDBRS]

@VladRassokhin Any answers?

[realrill]

For anyone with a similar problem, a workaround I found was to create a new vault connection with a different namespace for every secret I wanted to access.

Hi @rcohenDBRS, I have same issue.

I am trying to read one secret for test purpose but the property within the build script appears as a single string like %password%. Parameter namespace not empty in this scenario.

Without parameter namespace I get 403 error message within client log plus Failed to revoke token within server log.

@VladRassokhin any idea for this issue due to looks like it is not an unique problem.

Edit I have managed it and it works now. Would be really beneficial if someone could update the documentation!

[rcohenDBRS]

@realrill What did you manage?

[realrill]

@rcohenDBRS Reading (multiple) secrets, but only with your solution. I have had incorrect parameter, that was the source of my issue.

[pscheit]

same here. Can someone please explain the workaround in more detailed ways?