Token not renew after max-ttl reached

[vfouqueron]

Describe the bug When max-ttl is reached within vault, the token ttl is shortened (Vault indicate with a warning of the kind "TTL of "50s" exceeded the effective max_ttl of "40s"; TTL value is capped accordingly".

After that, the app tries once more to renew the token using "sys/leases/renew" path, but get an error this time as lease has expired and has been deleted (with error "lease renewal failed: lease_id=database/dev/trace/mariadb/creds/certification-readwrite/lxWq6KOj9PO67KIn3PnCxTi8 error="lease not found"). On Spring side, the stacktrace is:

WARN 1 --- [g-Cloud-Vault-2] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='database/dev/trace/mariadb/creds/certification-readwrite', mode=RENEW]] Lease [leaseId='database/dev/trace/mariadb/creds/certification-readwrite/lxWq6KOj9PO67KIn3PnCxTi8', leaseDuration=PT10S, renewable=true] Cannot renew lease: Status 400 Bad Request lease not found

org.springframework.vault.VaultException: Cannot renew lease: Status 400 Bad Request lease not found
    at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:670)
    at org.springframework.vault.core.lease.SecretLeaseContainer.renewAndSchedule(SecretLeaseContainer.java:553)
    at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRenewal$2(SecretLeaseContainer.java:545)
    at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:838)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)  
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.springframework.vault.VaultException: Status 400 Bad Request: lease not found
    at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:64)
    at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:451)
    at org.springframework.vault.core.lease.SecretLeaseContainer.doRenew(SecretLeaseContainer.java:705) 
    at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:646)
    ... 11 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{"errors":["lease not found"]}<EOL>"
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:915)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:864)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:764)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:646)
    at org.springframework.vault.core.lease.LeaseEndpoints.put(LeaseEndpoints.java:161)
    at org.springframework.vault.core.lease.LeaseEndpoints$2.renew(LeaseEndpoints.java:82)
    at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$doRenew$4(SecretLeaseContainer.java:705)
    at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:448)
    ... 13 common frames omitted

After that nothing more happens. From what I've read from different issues and from comment in log, Spring Vault is supposed to login again, but that's not what is happening in my case.

Sample We are in a Kubernetes cluster with a Vault and our app deployed inside. The Kubernetes token was well renewed inside the pod. The error is on mariadb database credentials access, the lease has expired and user is deleted from the database. So the app cannot access database anymore.

[mp911de]

Let's sort things a bit first. You're talking about reaching max-ttl and re-login.

Do I understand correctly that the login token has expired and that has revoked all associated leases?

If so, we addressed that scenario with #698. Right now, the fix is only available in the current release candidate which you could test against (4.1.0-RC1), the GA release follows by next week.

[vfouqueron]

That's a good question! I'm mixing in my head a little of all authentication kinds. Currently, I have no max-ttl on the kubernetes login. However, in my database secret, the role has a ttl of 1h and a max-ttl of 24h. I want my database credentials to be renewed every 24h, that's what I put that value. I expected Spring vault to create a new lease after 24h and not renew the current one. For the login, I will probably implement ttl and max-ttl once the 4.1.0 is out.

[mp911de]

If the expiry isn't caused by the expiring login, I strongly recommend using a higher min-renewal timeout. It defaults to 10 seconds but that can become an issue if GC and latency are involved.

As this ticket isn't actionable, I'm going to close it.

[vfouqueron]

@mp911de I’m sorry, I don’t understand you’re answer. Does it mean that the lib won’t support any other expiration than the login one? So the lease expiration won’t be supported ? Changing min-renewal won’t correct my issue if I understand well, it’s just a general advice about my setup. I don’t understand yet that part either, sorry

[mp911de]

Spring Vault considers a lease expired if the remaining lease time is below spring.cloud.vault.config.lifecycle.min-renewal (defaults to 10 seconds).

In your example above, 40 seconds were above the 10 second limit. But because the login token expired, the old version didn't re-request a new lease.

The upcoming 4.1.0 GA release will address the re-acquisition of a new lease.