Enhance Spring ConfigServer: Introduce {secret} Prefix for Efficient Management of Shared Secrets

[sergioasantiago]

Is your feature request related to a problem? Please describe.

Currently, when externalizing secrets from the git backend to a secret manager (such as Vault or AWS Secret Manager) in a Spring ConfigServer setup, the need to duplicate secrets across multiple microservices can be cumbersome. Each microservice requests its configuration from ConfigServer using the microservice name as the application, leading to redundancy when a secret is shared among several microservices.

Describe the solution you'd like

I would like to propose an enhancement that allows for more efficient management of shared secrets. Specifically, the ability to extend the {cipher} prefix to introduce a new prefix, let's call it {secret}, and include the path to the secret in the secret manager. For example: {secret}/secret/myservice/live:apipassword=test. This modification would enable the reuse of the same secret across multiple microservices, improving efficiency and easing the process of updating shared secrets during refactoring.

Describe alternatives you've considered

One alternative could be to continue with the current approach of duplicating secrets for each microservice, but this leads to redundancy, increased maintenance efforts, and potential issues during updates. The proposed {secret} prefix offers a more streamlined and scalable solution for managing shared secrets.

Additional context

Our infrastructure consists of approximately 500 microservices, and there are scenarios where a secret, such as an API password, needs to be shared among many microservices. The introduction of the {secret} prefix would not only enhance the flexibility of managing shared secrets but also contribute to a more efficient and scalable configuration setup for large microservices architectures.

Contributions

I am available and willing to create a pull request to implement this feature once the proposal is reviewed and accepted by the Spring ConfigServer project maintainers.

[ryanjbaxter]

The config server looks for something in the backend named application all properties within this store is served to all microservices. For example with Vault you could do

vault kv put secret/application foo=bar baz=bam

Every microservice that request configuration from the config server would then receive the foo and baz properties.

And with AWS Secret Manager

When using AWS Secrets Manager as a backend, you can share configuration with all applications by placing configuration in /application/ or by placing it in the default profile for the application.

Is this not what you are requesting?

[sergioasantiago]

Hi @ryanjbaxter, thank you for your prompt response.

While the solution you provided is appreciated, it doesn't precisely align with our requirements.

Our objective is to maintain a dual backend approach, utilizing git for non-sensitive data and vault for managing secrets. We employ a composite configuration to merge the outcomes seamlessly. The configuration snippet is as follows:

spring:
  cloud:
    config:
      server:
        encrypt:
          plainTextEncrypt: true
        composite:
          - type: git
            uri: git@mygithub.com:configuration/{application}.git
            default-label: master
            ignoreLocalSshSettings: true
          - type: git
            uri: git@mygithub.com:configuration/global.git
            default-label: master
            refreshRate: 86400
          - type: vault
            host: vault.myhost
            backend: configserver
            kvVersion: 2
            authentication: TOKEN

During startup, services request configurations using their service names (e.g., myservice). This retrieves keys from:

• mygithub.com:configuration/myservice.git
• mygithub.com:configuration/global.git
• vault: /configserver/myservice

This setup works seamlessly when secrets for myservice are located under /configserver/myservice. Our secret schema looks like this:

/configserver/rds/myservice/password
/configserver/apipassword/anotherservice

To enhance maintenance, we propose the ability to reference secrets in the git configuration, such as:

{vault}/configserver/apipassword/anotherservice

or a more generic form:

{secret}/configserver/apipassword/anotherservice

This would prompt the configserver to fetch the secret by path from the configured secret backend (in this example, Vault). This approach streamlines management, especially when identical secrets are used across multiple services, like /configserver/apipassword/anotherservice. Additionally, we can create namespaces like /configserver/rds to organize secrets efficiently.

I trust this clarifies our request. Feel free to reach out if you require additional details. Thanks.

[ryanjbaxter]

I am not sure I am understanding the problem....

It seems like if you structured your secrets like this:

/configserver/application/

And placed all common secrets under there all applications would get these properties.

However since you have chosen to structure your secrets differently you cannot do this?

I also don't understand why your git configuration would reference these common configurations.

Why wouldn't you set

configserver/apipassword apipasswordproperty=mypassword

in Vault?

Why would you set the property in Git and then reference the value in Vault?

[sergioasantiago]

Consider a scenario with three services: ServiceA, ServiceB, and ServiceC.

ServiceA needs to make requests to ServiceC, and ServiceB also needs to make requests to ServiceC. Both ServiceA and ServiceB require the apiPassword for ServiceC.

With the current configserver implementation, two separate secrets are needed in Vault:

/configserver/ServiceA serviceCApiPassword=mypassword

/configserver/ServiceB serviceCApiPassword=mypassword

Consequently, when ServiceA requests its configurations, it retrieves the apiPassword to connect to ServiceC, and the same process occurs for ServiceB.

The desired outcome is to streamline this process by having a single secret:

configserver/ServiceC apiPassword=mypassword

In the configurations for both ServiceA and ServiceB, we aim to reference the Vault path:

{vault}configserver/ServiceC

This approach eliminates the need for duplicated secrets. While this example highlights only two instances of duplicated secrets, envision scaling it up to a scenario involving 500 microservices that share common secrets. The goal is to enhance efficiency and simplify secret management across a large-scale microservices architecture.

[ryanjbaxter]

This can be accomplished with

/configserver/application serviceCApiPassword=mypassword

Both ServiceA and ServiceB will get serviceCApiPassword when fetching their configuration from the config server

[sergioasantiago]

Yes, indeed it can, but also a ServiceX would get this secret even though it is not needed, so that would be a security issue leaking secrets to all services.

[ryanjbaxter]

Ok so what you really want is a list of applications allowed or a list of applications not allowed?

that seems like a much simpler way to accomplish what you are looking for

[sergioasantiago]

I guess I know what you are about to suggest. The service sends the token and we link the secrets to the token sent by the service. Yes, that would definitely work. But we can't use client authentication in our use case (for many other reasons), we need to use server authentication (configserver authenticate to vault)

[ryanjbaxter]

Got it. I could imagine something on the config server that specify which application names will get the common configuration properties from application. We could also offer a property the does the opposite and instead is an exclusion list.

[sergioasantiago]

I'm not sure if I fully understand that. How that would look like?

[sergioasantiago]

I stumbled here by accident.

This looks like exactly what I planned to have. So I need to enable the vault profile and set the value starting with {vault}.

[sergioasantiago]

I just did some tests and it does exactly what is needed for this use case.

• I needed to set the configuration for Vault under spring, not under composite.

  spring:
  cloud:
  config:
    server:
      vault:
        backend: configserver
        skipSslValidation: true
        kvVersion: 2
        authentication: TOKEN
        token: "mytoken"

• I need to enable the vault profile
• I need to use the following pattern to set the key: one: "{vault}:rds/testdatabase#password"
• My secret is stored in vault like this:

So in the end, it is a matter of lack of documentation since I could not find any mention about this feature in spring cloud docs.

[ryanjbaxter]

Huh I didnt even know about that. Would you be interested in adding some docs on this?

[sergioasantiago]

Sure, I can take a look. Which section do you suggest to put this?

[ryanjbaxter]

I would say here https://github.com/spring-cloud/spring-cloud-config/blob/4.0.x/docs/src/main/asciidoc/spring-cloud-config.adoc#vault-backend

[pukkaone]

There is already issue #1785 to document the {vault} decryptor.