Explicit token renewal when authentication with current token fails

[venkateshdomakonda]

Hello,

I am using the CryptoConfiguration class, which extends AbstractVaultConfiguration to perform crypto operations.

@Configuration
public class CryptoConfiguration extends AbstractVaultConfiguration {
  private static final String ROOT_CERT = "/tmp/cacertbundle.jks";

  @Override
  public ClientAuthentication clientAuthentication() {
    KubernetesAuthenticationOptions options = KubernetesAuthenticationOptions.builder().role("key-role")
        .build();
    return new KubernetesAuthentication(options, restOperations());
  }

  @Override
  public VaultEndpoint vaultEndpoint() {
    VaultEndpoint endpoint = VaultEndpoint.create(key - management, 8200);
    endpoint.setScheme("https");
    return endpoint;
  }

  @Override
  @Bean
  public ClientFactoryWrapper clientHttpRequestFactoryWrapper() {
    char[] password = {};
    SslConfiguration sslConfig = SslConfiguration.forTrustStore(new FileSystemResource(ROOT_CERT),
        password);
    return new ClientFactoryWrapper(ClientHttpRequestFactoryFactory.create(clientOptions(), sslConfig));
  }
}

public class CryptoAgent implements CryptoAgentInterface {
  private static final String CRYPTO_KEY_NAME = "cm-key-v1";
  private static final Retrier retrier = new Retrier();
  private VaultTemplate vaultTemplate;
  private VaultTransitOperations transitOps;
  private AnnotationConfigApplicationContext ctx;

  /**
  * CryptoConfiguration is used as configuration for vault agent.
  */
  public CryptoAgent() {
    ctx = new AnnotationConfigApplicationContext();
    ctx.register(CryptoConfiguration.class);
    ctx.refresh();
    this.vaultTemplate = ctx.getBean(VaultTemplate.class);
    this.transitOps = vaultTemplate.opsForTransit();
  }

  @Override
  public String encrypt(String input) {
    return retrier.runtimeRetry("encrypt", () -> {
      return transitOps.encrypt(CRYPTO_KEY_NAME, Plaintext.of(input)).getCiphertext();
    }, "Failed to encrypt");
  }

  @Override
  public String decrypt(String input) {
    return retrier.runtimeRetry("decrypt", () -> {
      return transitOps.decrypt(CRYPTO_KEY_NAME, Ciphertext.of(input)).asString();
    }, "Failed to decrypt");
  }

  private void handleVaultException(VaultException e) {
    if (VaultUtils.recreateVaultTemplate(e)) {
      recreateVaultTemplate();
    }
  }

  private void recreateVaultTemplate() {
         synchronized (this) {
             LOGGER.info("Recreate crypto agent vault template");
             // Close the existing context
             ctx.close();
             ctx = new AnnotationConfigApplicationContext();
             ctx.register(VaultCryptoConfiguration.class);
             ctx.refresh();
             this.vaultTemplate = ctx.getBean(VaultTemplate.class);
             this.transitOps = vaultTemplate.opsForTransit();
         }
     }

For errors like 'Vault Operation failed. Reason: Status 403 Forbidden or 401 Unauthorized', I am planning to recreate the VaultTemplate. I guess this is not good approach to handle 403/401 during encrypt/decrypt operations.

Could you please advise on the recommended approach to renew the token explicitly when spring vault failed to do it?

[mp911de]

Which version of Spring Vault do you use? AbstractVaultConfiguration uses LifecycleAwareSessionManager that renews login tokens until they reach max_ttl. After reaching max_ttl, Spring Vault logs in again using the configured ClientAuthentication.

Have you tried increasing the log level for org.springframework.vault.authentication to INFO?

Upon a vault access error, you should check whether the login token has been rotated, renewed and what the cause for 401/403 responses is.

[venkateshdomakonda]

Hello,

Thanks for your response.

We are using below dependencies for spring vault client.

      <dependency>
        <groupId>org.springframework.vault</groupId>
        <artifactId>spring-vault-core</artifactId>
        <version>3.1.1</version>
      </dependency>
      <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-core</artifactId>
        <version>6.1.6</version>
      </dependency>
      <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-web</artifactId>
        <version>6.1.6</version>
      </dependency>
      <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-beans</artifactId>
        <version>6.1.6</version>
      </dependency>
      <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-context</artifactId>
        <version>6.1.6</version>
      </dependency>

Unfortunately INFO level is not set on 3pp's that are used in our application. I see only the below stacktrace for 403 error. Our application has two vault clients 1. cryptoClient and 2.secretStorageClient with similar setup.

SecurityRetrier [] - Vault Operation failed. Reason: Status 403 Forbidden [service-credentials]: permission denied, Exception: org.springframework.vault.VaultException: Status 403 Forbidden [service-credentials]: permission denied\n\tat org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:84)\n\tat org.springframework.vault.core.VaultVersionedKeyValueTemplate.lambda$doRead$0(VaultVersionedKeyValueTemplate.java:109)\n\tat org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:451)\n\tat org.springframework.vault.core.VaultVersionedKeyValueTemplate.doRead(VaultVersionedKeyValueTemplate.java:94)\n\tat org.springframework.vault.core.VaultVersionedKeyValueTemplate.get(VaultVersionedKeyValueTemplate.java:85)\n\tat org.springframework.vault.core.VaultVersionedKeyValueOperations.get(VaultVersionedKeyValueOperations.java:70)\n\tat security.VaultKeyStorageAgent.lambda$getKey$2(VaultKeyStorageAgent.java:94)\n\tat security.SecurityRetrier.retry(SecurityRetrier.java:46)\n\tat 
...
...
jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)\n\tat com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:142)\n\tat com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: \"{\"errors\":[\"permission denied\"]}<EOL>\"\n\tat org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109)\n\tat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)\n\tat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)\n\tat org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)\n\tat org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942)\n\tat org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891)\n\tat org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790)\n\tat org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:672)\n\tat org.springframework.vault.core.VaultVersionedKeyValueTemplate.lambda$doRead$0(VaultVersionedKeyValueTemplate.java:97)\n\t... 24 more\n"}

At the moment after retries for 1 minute, we trigger restart of the application as recovery measure and wanted to avoid it now.

[mp911de]

Without further information, there's not much we can do here. You really need to investigate what happens with the token before and what happens with renewal.

[venkateshdomakonda]

Currently, obtaining the desired logs is challenging for me. I understand that the recovery solution I proposed may not be optimal. However, do you anticipate any issues with it?

[mp911de]

Restarting the context is similar to dropping a token from the session manager. So I generally wonder why one variant works while the other doesn't.

I am closing this ticket as it isn't actionable. If you would like us to look at this issue, please provide additional information, and we will re-open it.