Spring-Cloud-Config-Server with Vault Backend and Vault Client at the same time causes issue with X-VAULT-TOKEN

[tuxdevelop]

Spring Cloud Version: Hoxton.SR3 / Hoxton.SR4 Spring Boot: 2.2.7.RELEASE

Hi, we are using the Spring Cloud Config server with a vault and git backend. The config server itself uses spring cloud vault to fetch its own secrets via vault to connect to the git backend.

       <!-- spring cloud config -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>

         <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-vault-config</artifactId>
        </dependency>

Our clients are sending the X-CONFIG-TOKEN header to the config server and this will be sent via config-server to Vault (X-VAULT-TOKEN).

The tokens are generated dynamically for each deployment based on application policies. Each token can only access an application/deployment specific path in vault.

policy for applicationA

path "secret/applicationA" {
  capabilities = ["read"]
}

path "secret/applicationA/*" {
  capabilities = ["read"]
}

policy for applicationB

path "secret/applicationB" {
  capabilities = ["read"]
}

path "secret/applicationB/*" {
  capabilities = ["read"]
}

The tokens are not renewable and they have limited ttl.

This worked pretty well until we tried to upgrade from the Greenwich.SR5 release train to Hoxton.SR3. (It is the same behavior with Hoxton.SR4).

After the upgrade we started to see 403 response codes from vault during deployments. After a rollback of the config server to Greenwich, the same deployments went through without any problems.

org.springframework.web.client.HttpClientErrorException$Forbidden: 403 : [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
]
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:112)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:612)
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$1(VaultKeyValueAccessor.java:133)
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$2(VaultKeyValueAccessor.java:168)
        at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:388)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:165)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:132)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:109)
        at org.springframework.vault.core.VaultKeyValue1Template.get(VaultKeyValue1Template.java:69)
        at org.springframework.cloud.config.server.environment.vault.SpringVaultEnvironmentRepository.read(SpringVaultEnvironmentRepository.java:51)

A debugging session showed the issue:

The config server reuses an old X-CONFIG-TOKEN of a former client request (cached?). Our suspect is the SessionManager in the VaultTemplate which keeps/injects the wrong header value.

After a fresh deployment of the config server itself, the behavior is the following

• deployment of applicationA with X-CONFIG-TOKEN "A" -> successful

• deployment of applicationB with X-CONFIG-TOKEN "B" -> failed/403 because X-CONFIG-TOKEN "A" is used by the config server

• for each deployment afterwards, token "A" is used

Our configuration looks like this:

spring:
  profiles:
    active: "git,vault"
  cloud:
    config:
      server:
        vault:
          order: 1
          host: ${VAULT_HOST}
          port: 443
          scheme: https
          profile-separator: "/"
          kv-version: 1
        git:
          order: 2
          uri: ${GIT_CONFIG_URI}
          clone-on-start: true
          force-pull: true
          search-paths: {application}
      enabled: false
      discovery:
        enabled: false
    vault:
      enabled: true
      token: "${VAULT_TOKEN}"
      host: ${VAULT_HOST}
      port: 443
      scheme: https
      authentication: token
      kv:
        backend-version: 1 

A downgrade to Hoxton.SR2 also let this issue disappear.

Can you please have a look why the config-server behaves in a way like this for >= Hoxton.SR3?

A sample project which shows the issue is available at https://github.com/tuxdevelop/spring-cloud-config-issue1622

Thank you and greetings,

Marcel

[spencergibb]

@mp911de any thoughts?

[mp911de]

Config server has switched from a custom Vault integration to Spring Cloud Vault which uses Spring Vault beans to configure the Vault integration and to interact with Vault.

With this change, the config server uses LifecycleAwareSessionManager that caches a previously obtained Vault token. While this is desirable for authentication methods (app role, TLS authentication, …), it's not so much desirable for the per-request token retrieval that config server uses to obtain the token from incoming config server requests.

We would need to add a pass-thru SessionManager implementation for Spring Cloud Config if the Vault token is retrieved from incoming HTTP requests to prevent caching.

[tuxdevelop]

Hi @mp911de, is there a chance that the Hoxton release train will get a SessionManager which is supporting this? We are currently using Spring Boot 2.1.x and Spring Cloud Greenwich. Spring Boot 2.1.x will be end of life at the end of the year.

Thanks,

Marcel

[marbon87]

Hey @tuxdevelop, how did you work around this issue?

[tuxdevelop]

Hi @marbon87, the workaround was to use spring-cloud-vault (in the apps directly) next to the config-server to access vault and use the config-server for all non vault configuration properties. This was the only solutions which helped us to continue using our setup.

[marbon87]

Hm, that's sad because that requires spring-vault-core as a dependency in every application.

The problem with a custom implementation of SessionManager is that i don't see any place to inject it into VaultTemplate. The LifecycleAwareSessionManager is defined in AbstractVaultConfiguration without any possibility to override it. SpringVaultClientConfiguration, that extends AbstractVaultConfiguration, is imported by EnvironmentRepositoryConfiguration and there is no way to customize it.

@mp911de am i missing something? Is there a possibility to configuare a custom SessionManager?

[marbon87]

Finally i got it working with this (ugly) solution:

@Bean
public BeanPostProcessor sessionManagerBeanPostProcessor(AbstractVaultConfiguration vaultConfiguration) {
    return new BeanPostProcessor() {
        @Override
        public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
            if (SessionManager.class.isInstance(bean)) {
                return new NoSessionManager(vaultConfiguration.clientAuthentication());
            }
            return bean;
        }
    };
}

public class NoSessionManager implements SessionManager {

    private final ClientAuthentication clientAuthentication;

    public NoSessionManager(ClientAuthentication clientAuthentication) {
        this.clientAuthentication = clientAuthentication;
    }

    @Override
    public VaultToken getSessionToken() {
        return clientAuthentication.login();
    }
}

[vazuev]

Hello @ryanjbaxter @spencergibb ! Faced this issue today. Could you prioritise this?