search

spring-cloud / spring-cloud-vault

Configuration Integration with HashiCorp Vault
http://cloud.spring.io/spring-cloud-vault/
Apache License 2.0
274 stars 152 forks source link

Endless 403 forbidden #548

Closed Vity01 closed 3 years ago

Vity01 commented 3 years ago

Describe the bug It looks like whenever I start to block the Vault server (or there is any network problem) for a specific time, the vault client always gets a return code 403 from the server, the token is never refreshed and the client is not able to recover from this state. Only the whole app restart fixes the problem. I am not able to find another workaround. I am able to reproduce problem with our specific Vault configuration locally - very short Token timeout and other stuff. It applies for spring-cloud-starter-vault-config:2.2.5. The whole set up is quite complicated, I can demonstrate it on Zoom call with proxying calls

mp911de commented 3 years ago

If you would like us to spend some time helping you to diagnose the problem, please spend some time describing it and, ideally, providing a minimal sample that reproduces the problem.

Vity01 commented 3 years ago

The whole test env is ready on my comp and it's dependent on the Vault settings. I am not familiar with the Vault set up.. etc. If you find 10 minutes , I can show you there is nothing special in spring-vault usage and the ugly bug is sitting there.

mp911de commented 3 years ago

That should be fixed with https://github.com/spring-projects/spring-vault/issues/601. Can you upgrade to Spring Vault 2.2.3.RELEASE and retest?

Vity01 commented 3 years ago

I am a bit confused about the artifacts and the versioning now. I am using org.springframework.cloud:spring-cloud-starter-vault-config:2.2.6.RELEASE which uses spring-vault-core:2.2.2.RELEASE (max version) Is spring-cloud-starter-vault-config supposed to be obsolete? Which artifact should I try?

mp911de commented 3 years ago

Spring Cloud Vault builds on top of Spring Vault that ships the core functionality. You can stick with your Spring Cloud Vault version, just upgrade Spring Vault to 2.2.3.RELEASE.

Vity01 commented 3 years ago

I am sorry, but I don't see any spring-vault artifact having 2.2.3.RELEASE version https://search.maven.org/search?q=spring%20vault

EDIT: It appeared there 1 minute ago

mp911de commented 3 years ago

Mind the caches. https://repo1.maven.org/maven2/org/springframework/vault/spring-vault-core/ is likely the more reliable source.

Vity01 commented 3 years ago

Unfortunately, the bug seems to persist. I am still able to reproduce it. Artifacts used: image I am ready to show it to you, setting whole the project would be too extreme.. I am sure you recognize the problem in a minute (or you find much easier way to reproduce it locally)

mp911de commented 3 years ago

We can't always jump onto calls with every individual reporter as we're pretty busy already. Having a reproducer available allows us to investigate the issue once we have some time to look into it.

Vity01 commented 3 years ago

I agree on reproducible test, but this is pretty complex and it would take me (with my bare Hashicorp Vault knowledge) several days to come with something (believe me , I tried that). With your knowledge it can be just a few minutes. I've already spent days to be able to find it and reproduce it...

mp911de commented 3 years ago

We've documented ACL requirements via #561.

rhigonet commented 2 years ago

It is still a thing? We found exactly the same issue in our PROD environment. The application was working fine, and some of the Vault instances were restarted. Then the application started an endless loop of 403 on the requests to the vault. Only the whole app restart fixed the issue.

We couldn't reproduce it yet

Vity01 commented 2 years ago

Try to reproduce it this way - try to use some proxy for connections and then kill the proxy for some time