posts/hashicorp-vault/rotate-dynamic-relational-database-connection-in-spring-at-runtime/

[utterances-bot]

Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring · Secrets as a Service

Rotate Expiring Spring Cloud Vault Database Credentials Without Downtime

https://secrets-as-a-service.com/posts/hashicorp-vault/rotate-dynamic-relational-database-connection-in-spring-at-runtime/

[usr42]

Author here. I am looking forward to feedback and comments!

[marcino239]

Good read. Thx!

[usr42]

@marcino239 Thanks for the feedback. If you have any questions or if anything is not completely sure, please let me know. Do the Kotlin examples work for you or would you be interested in a Java version of this blog post?

[marcino239]

Hi @usr42 - Kotlin works well here - its less verbose than Java, but that's personal preference. One idea for a blog post if I may suggest would be to show full end to end simple spring app setup into docker image and proper security (i.e. passwords in secrets file or some other secret manager like above). Majority of blog posts put out there a lot of code rather then focusing what is important.

Great choice of images!

[ncorrare]

This is great. Just be aware that while the format of the lease object is pretty much static (with the exception of PKI if I remember correctly), the secret object varies from engine to engine.

[dheerajjoshim]

This looks promising. However I have few queries on handling Hikari pool under load.

I would assume when lease is expired and mode is renew, vault system has already revoked the secrets and presumably removed the username and password from database. Now as we process the event and try to change the username and password in our pool, any outgoing connection will still try to use old username and password and connection would fail.

So we may need to handle retries for access denied cases? Any thoughts?

[JahnaviMolleti]

This article is very helpful. I implemented same using spring cloud vault and postgres but i face issue where after SecretLeaseExpiredEvent when we try to rotate credentials we get 403 error on rest call to "GET /v1/database/creds/role HTTP/1.1[\r][\n]" endpoint (org.springframework.vault.VaultException: Status 403 Forbidden [database/creds/role]: permission denied). Do you have any idea about this issue?

[thakur-mohit]

Kindly share the java implementation as well. @Jahnavi Are you able to resolve the issue. If possible, kindly place code on github and share the url.

[JahnaviMolleti]

@thakur-mohit Yes we resolved the issue. The 403 error is because of X-Vault-Token getting expired at the same time while we try to fetch new database credentials using rotation. We increased the ttl of our X-Vault-Token fixed the issue. But the database credentials rotation with postgres has another issue where i need to modify ownership of all the schema to new role as the foreign-key relation fails when different roles act as owners. Even if we write our role.sql script to update ownership to new role it do not support multiple instances of database connecting at same time. I plan to migrate to static role with password rotation.

[thakur-mohit]

@Jahnavi In our case, we don't have dynamic secrets. Also , we are using hikaricp. When we deploy on openshift, we are able to fetch secret from Vault. We are not using database engine but Key/Value. At the time of appln start up , I'm able to read secrets from Vault. But hikaricp keep on adding stats.

Below is the VaultConfig class.

import java.util.Map;

import javax.annotation.PostConstruct;

import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.annotation.Configuration; import org.springframework.vault.core.VaultOperations; import org.springframework.vault.core.lease.SecretLeaseContainer; import org.springframework.vault.core.lease.domain.RequestedSecret.Mode; import org.springframework.vault.core.lease.event.LeaseListenerAdapter; import org.springframework.vault.core.lease.event.SecretLeaseCreatedEvent; import org.springframework.vault.core.lease.event.SecretLeaseEvent; import org.springframework.vault.core.lease.event.SecretLeaseExpiredEvent;

import com.zaxxer.hikari.HikariDataSource;

import lombok.extern.slf4j.Slf4j;

@ConditionalOnBean(SecretLeaseContainer.class) @Configuration @Slf4j public class VaultDatabaseConfig {

  @Autowired
  private ConfigurableApplicationContext applicationContext;
  @Autowired
  private HikariDataSource hikariDataSource;
  @Autowired
  private SecretLeaseContainer leaseContainer;
  @Value("${env}")
  private  String env;
  @Value("${vault.enabled}")
  private  boolean vaultEnabled;
  @Value("${spring.datasource.username}")
  private  String username;
  @Value("${spring.datasource.password}")
  private  String password;

@PostConstruct
private void postConstruct() {
    if(vaultEnabled) {
    String vaultCredsPath = "kv/team/openshift/"+env;

    leaseContainer.addLeaseListener(new LeaseListenerAdapter() {
        @Override
        public void onLeaseEvent(SecretLeaseEvent event) {

            log.info("Lease change for DB: "+ event.toString());
            log.info("Paths: "+ (event.getSource().getPath().equals(vaultCredsPath)));
                if (vaultCredsPath.equals(event.getSource().getPath())) {
                    log.info("event lease : "+event.getLease().toString());
                        if (event instanceof SecretLeaseCreatedEvent && event.getSource().getMode().equals(Mode.ROTATE)) {
                            log.info("Lease change SecretLeaseCreatedEvent: "+ event.toString());
                            String pwd = password;

                            if(pwd.isEmpty()) {
                                 log.error ( "Cannot get updated DB credentials. Shutting down.");
                                 applicationContext.close();

                            }
                            refreshDatabaseConnection(pwd);
                        }

                        if (event instanceof SecretLeaseExpiredEvent && leaseContainer..equals(Mode.RENEW)) {
                            log.info("Lease change SecretLeaseExpiredEvent: "+ event.toString());
                            leaseContainer.requestRotatingSecret(vaultCredsPath) ;
                        }
                }
        }

    });
    }
}

protected void refreshDatabaseConnection(String pwd) {
    updateDbProperties(pwd);
    updateDataSource(pwd);

}

private void updateDataSource(String pwd) {
    log.info("update data source: "+ pwd);
     hikariDataSource.getHikariConfigMXBean().setPassword(pwd);
     if(null!=hikariDataSource.getHikariPoolMXBean()) {
         hikariDataSource.getHikariPoolMXBean().softEvictConnections();
     }
}

private void updateDbProperties(String pwd) {
    log.info("update data source: "+ pwd);
    System.setProperty("spring.datasource.password", pwd);

}

}

Below is the application.yml.

spring: datasource: hikari: connection-timeout: 70000 maximum-pool-size: 2 minimum-idle: 1 register-mbeans: true url: jdbc:mysql://${DB_URL}:${DB_PORT}/${DB_NAME}?serverTimezone=UTC username: ${DB_USERNAME} password: ${DB_PASSWORD}

[thakur-mohit]

@jahnavi Below are the logs

0-12-07 20:57:15.891 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@3700ffd: (connection evicted) 2020-12-07 20:57:15.910 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@58e6c6e3 2020-12-07 20:57:15.910 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After adding stats (total=1,

After sometime healthcheck fails and appln shutdown.

[jesantana]

Have some one achieved the same, but with Cassandra accessed via DataStax java driver?

I created this post in StackOverflow asking about this https://stackoverflow.com/questions/68320750/change-datastax-java-driver-user-and-password-in-runtime

[sullrich84]

Has anyone implemented this fix for MongoDB connection? I have also asked this on SO: https://stackoverflow.com/questions/73408693/update-mongoclient-credentials-at-runtime

[pradeepg-alkira]

Does hosted vault support this?

[Archanashetty7]

Hi Everyone, lease does not rotate if @EnableConfigServer annotation is used in application ,any suggestions here