Database rotation causes VaultException "no lease found"

YahavGB commented 3 years ago

Hi!

Describe the bug I've started to integrate the HashiCorp Vault with my Spring-based project. My Vault configuration:

Enabled AppRole, database integration and MySQL.
Created an AppRole user & policy. The policy defined as
```
path "database/creds/app" {
policy = "read"
}
```

path "sys/renew/database/creds/*" { capabilities = ["read", "update", "list"] }

path "sys/leases/*" { capabilities = ["create", "read", "update", "delete", "list"] }

path "auth/token/renew-self" { capabilities = ["read", "update", "list"] }


In Spring, I defined Vault using `boostrap.properties` as follows:

spring: application: name: app cloud: vault: application-name: app authentication: APPROLE host: ${VAULT_HOST:localhost} port: ${VAULT_PORT:8200} app-role: role-id: ${VAULT_APP_ROLE_ID} secret-id: ${VAULT_APP_ROLE_SECRET_ID} scheme: http database: enabled: true role: db-role kv: enabled: true default-context: app application-name: app


Then, in `application.yaml`, I defined:

spring: datasource: url: jdbc:mysql://${DB_HOST:localhost}:${DB_PORT:3306}/${DB_NAME}?characterEncoding=UTF-8&serverTimezone=UTC driver-class-name: com.mysql.cj.jdbc.Driver


When starting the application, it's being able to authenticate successfully with AppRole and generate a custom database credentials. I was also being able to verify it with MySQL.

My problem begins when trying to rotate the DB credentials: I understood that Spring doesn't support the rotate policy by default, so following [This tutorial](https://github.com/ivangfr/springboot-vault-examples), I created the following class:

@Slf4j @RequiredArgsConstructor @Configuration public class VaultLeaseConfig { @Value("${spring.cloud.vault.database.role}") private String databaseRole;

private final ApplicationContext applicationContext; private final SecretLeaseContainer leaseContainer;

@PostConstruct private void postConstruct() { final String vaultCredsPath = String.format("database/creds/%s", databaseRole);

leaseContainer.addLeaseListener(event -> {
  LOG.info("==> Received event: {}", event);

  if (vaultCredsPath.equals(event.getSource().getPath())) {
    if (event instanceof SecretLeaseExpiredEvent &&
      event.getSource().getMode() == RequestedSecret.Mode.RENEW) {
      LOG.info("==> Replace RENEW lease by a ROTATE one.");
      leaseContainer.requestRotatingSecret(vaultCredsPath);
    } else if (event instanceof SecretLeaseCreatedEvent && event.getSource().getMode() == RequestedSecret.Mode.ROTATE) {
      SecretLeaseCreatedEvent secretLeaseCreatedEvent = (SecretLeaseCreatedEvent) event;
      String username = (String) secretLeaseCreatedEvent.getSecrets().get("username");
      String password = (String) secretLeaseCreatedEvent.getSecrets().get("password");

      LOG.info("==> Update System properties username & password");
      System.setProperty("spring.datasource.username", username);
      System.setProperty("spring.datasource.password", password);

      LOG.info("==> spring.datasource.username: {}", username);

      updateDataSource(username, password);
      LOG.info("==> DONE updateDataSource");
    }

    LOG.info("==> DONE HANDLE event: {}", event);
  }
});

}

private void updateDataSource(String username, String password) { HikariDataSource hikariDataSource = (HikariDataSource) applicationContext.getBean("dataSource");

LOG.info("==> Soft evict database connections");
HikariPoolMXBean hikariPoolMXBean = hikariDataSource.getHikariPoolMXBean();
if (hikariPoolMXBean != null) {
  hikariPoolMXBean.softEvictConnections();
}

LOG.info("==> Update database credentials");
HikariConfigMXBean hikariConfigMXBean = hikariDataSource.getHikariConfigMXBean();
hikariConfigMXBean.setUsername(username);
hikariConfigMXBean.setPassword(password);

}


AFAIK, the life-cycle is as follows:
- Spring uses the RENEW policy by default. Each time it'll get to the event listener in this class.
- If the lease can be renewed - it will and no condition will get satisifed.
- If the lease can not be renewed (a.k.a., expired) - then `event instanceof SecretLeaseExpiredEven` will be true and thus we will ask for a rotation.
- In such case, we will get a callback into the second condition, so that `event instanceof SecretLeaseCreatedEvent && event.getSource().getMode() == RequestedSecret.Mode.ROTATE`. In such case, we attempt to update Hikari credentials with those that we got.

However, even though I request a new, rotated, credentials - I still get an exception which yields that "no lease found" because of the previous, expired, credential:

2020-10-16 23:46:44.404 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> Received event: org.springframework.vault.core.lease.event.SecretLeaseCreatedEvent[source=RequestedSecret [path='database/creds/db-role', mode=ROTATE]] 2020-10-16 23:46:45.088 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> Update System properties username & password 2020-10-16 23:46:45.088 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> spring.datasource.username: v-app-f04R8KXmVMQ 2020-10-16 23:46:45.088 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> Soft evict database connections 2020-10-16 23:46:45.088 DEBUG [app,,,] 64810 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : Hikari - Closing connection com.mysql.cj.jdbc.ConnectionImpl@59d9d656: (connection evicted) 2020-10-16 23:46:45.089 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> Update database credentials 2020-10-16 23:46:45.089 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> DONE updateDataSource 2020-10-16 23:46:45.089 INFO [app,,,] 64810 --- [g-Cloud-Vault-1] i.p.h.a.configurations.VaultLeaseConfig : ==> DONE HANDLE event: org.springframework.vault.core.lease.event.SecretLeaseCreatedEvent[source=RequestedSecret [path='database/creds/db-role', mode=ROTATE]] 2020-10-16 23:46:45.090 WARN [app,,,] 64810 --- [g-Cloud-Vault-1] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='database/creds/db-role', mode=ROTATE]] Lease [leaseId='database/creds/db-role/qWnW8gCOCQjnmoFHU8I8sdlp', leaseDuration=PT10S, renewable=true] Cannot renew lease: Status 400 Bad Requestlease not found; nested exception is org.springframework.vault.VaultException: Status 400 Bad Request: lease not found; nested exception is org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"errors":["lease not found"]} ]

org.springframework.vault.VaultException: Cannot renew lease: Status 400 Bad Requestlease not found; nested exception is org.springframework.vault.VaultException: Status 400 Bad Request: lease not found; nested exception is org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"errors":["lease not found"]} ] at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:713) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer.renewAndSchedule(SecretLeaseContainer.java:589) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRenewal$0(SecretLeaseContainer.java:581) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:891) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: org.springframework.vault.VaultException: Status 400 Bad Request: lease not found; nested exception is org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"errors":["lease not found"]} ] at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:63) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:391) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer.doRenew(SecretLeaseContainer.java:751) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:688) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] ... 12 common frames omitted Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"errors":["lease not found"]} ] at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:184) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:583) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.vault.core.lease.LeaseEndpoints$1.renew(LeaseEndpoints.java:59) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$doRenew$2(SecretLeaseContainer.java:752) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:388) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE] ... 14 common frames omitted



After digging around, I found out that a VaultException [is being thrown in such a case](https://github.com/spring-projects/spring-vault/blob/master/spring-vault-core/src/main/java/org/springframework/vault/core/lease/SecretLeaseContainer.java#L679) (it's actually happening at onError)- which is as far as I understand unexpected.

Will be glad to know if it's a bug or something that I'm missing!
Thanks!

YahavGB commented 3 years ago

After some more debugging, I could get rid of this log by doing

    leaseContainer.removeLeaseErrorListener(SecretLeaseEventPublisher.LoggingErrorListener.INSTANCE);

That's being said, I still doubt that it should be the default behavior, as the rotation of an expired key should be transparent and shouldn't raise an error. WDYT?

mp911de commented 3 years ago

Spring Vault supports credential rotation, it's the current relationship between Spring Boot and connection resources that does not support rotation for all supported configurations. Manual rotation is fine if you're propagating credentials yourself to the target (connection pool, driver, …).

lease not found should actually not happen, it should be guarded by the min-renewal threshold (defaulting to 10 seconds) that prevents short-lived leases from being renewed. Since your lease duration is 10 seconds, you get hit by the expiry threshold underrun. Spring Vault has no chance to renew the lease because the default is higher than your lease duration.

So from that perspective, the reported log event is perfectly fine because it hints you at a problem that can be solved by increasing the lease duration for the configured role.

spring-cloud / spring-cloud-vault

Database rotation causes VaultException "no lease found" #511