Spring Cloud Config Shared Configuration

dlaidlaw commented 7 years ago

With Spring Cloud Config, it is possible to have shared configuration. See: http://cloud.spring.io/spring-cloud-static/spring-cloud-config/1.3.3.RELEASE/single/spring-cloud-config.html#_sharing_configuration_with_all_applications

With this, the application properties and yaml files (application.properties, application.yml) are shared across all applications. Application-specific properties and yaml files (file name is ${spring.application.name}.yml or properties, along with their profile-specific names can override anything in the application.yml or properties.

The precedence order, from highest to lowest:

${spring.application.name}-${profile}.yml in the order of the active profiles
application-${profile}.yml in the config server
application.yml in config server
configurations stored in jar in their usual order. Note that the above uses the .yml file extension, but the .properties file extension is also valid.

This allows for setting application defaults while allowing application-specific overrides. It would be very convenient to support this type of configuration hierarchy with Kubernetes as well.

Can this type of thing be supported?

A configmap named application.yml could certainly be created. And configmaps named for each ${spring.application.name}.yml could also be created. Profile-specific configmaps named application-${profile}.yml and ${spring.application.name}-${profile}.yml are also easily created. All that is needed is for spring-cloud-kubernetes-config to load them all and merge them in the correct order.

Note, this is related to #129 and could solve that issue by providing the additional configurations in application-${profile}.yml configmaps where there is a profile for rabbitmq, etc.

deepank-sharma commented 6 years ago

We are also trying to find a parallel of Spring Cloud Config Server where we were able to read different application ymls corresponding to different spring profiles. Any solution to resolve this for multiple environments would be very useful. If this can be tied to spring.profiles.active would be the best solution.

spencergibb commented 6 years ago

Config Server can read from multiple profiles at once

dlaidlaw commented 6 years ago

@spencergibb Any doc on how to do this? The doc does not explain how multiple ConfigMaps could be targeted by multiple profiles being active.

Or is it required to have all of the profiles defined in the same ConfigMap? This is a less convenient setup. Having the ability to have a ConfigMap per profile is extremely useful for version control of configuration. This is supported by Spring Cloud Config.

Thanks! -Don

rikgig commented 6 years ago

It seems like in the Kubernetes environment, the profile value that we usually set in our Springboot applications gets replaced by the namespace. Is there a way to acheive what @dlaidlaw said and have shared profiles, like for examples a datasource definition, loaded in a Kubernetes configMap and have that profile loaded by still using the Spring Profiles capabilities to add or override configuration values based on your environment?

Thx for any pointers. -Eric.

ilseh commented 5 years ago

I was looking for a solution to use multiple configmaps with spring-cloud-starter-kubernetes-config and saw this issue. I solved it by using the spring.cloud.kubernetes.config.sources as described in https://cloud.spring.io/spring-cloud-static/spring-cloud-kubernetes/1.0.0.M2/multi/multi__configmap_propertysource.html

dlaidlaw commented 5 years ago

Oh nice!

ahoehma commented 3 months ago

Hi all,

for me it's still not 100% clear how to implement "hierarchical" configurations with spring cloud kubernetes config server.

is there anywhere a real example of such setups?

what I'm searching

top-level configmap in mynamespace with very global properties for ALL my services in the namespace - how to name this configmap?
for a group of my services I need also some "global" properties - how to name this configmap?
per application it's clear - I name the configmaps -default.

Would it may be possible just using "labels" (the primary concept in k8s) for selecting the right configmap(s) also in spring cloud config server kubernetes?

Examples what I need:

SPRING_CONFIG_ADDITIONAL_BY_LABEL=global, foo

and then config server is searching for more config-map's in my namespace labeled with "global" or "foo" and merge them together with the default configmap -.

Or is this may something to archive with just multiple "profiles"?

ryanjbaxter commented 3 months ago

Or is this may something to archive with just multiple "profiles"?

Yes I think that would be a better solution

top-level configmap in mynamespace with very global properties for ALL my services in the namespace - how to name this configmap?

The config map name should be application this will be included in all requests to the config server

for a group of my services I need also some "global" properties - how to name this configmap?

This is where profiles would come in. Pick a profile name for the group of applications and then create a config named application-[profile_name].

per application it's clear - I name the configmaps -default.

You can drop the -default it is not necessary.

ahoehma commented 3 months ago

@ryanjbaxter I tried but it's not working :-(

removed the "default" profile from my configmap's
i added a "application-brain-member" configmap (global settings for all the apps with active profile "brain-member"
i added a "cc-brain" configmap (settings only for the service cc-brain)

all in the same namespace

then I started a "springcloud/spring-cloud-kubernetes-configserver:3.1.3"

I checked the configserver:

http://localhost:8888/application/brain-member got only empty stuff stuff ... no properties

{
  "name": "application",
  "profiles": [
    "brain-member"
  ],
  "label": null,
  "version": null,
  "state": null,
  "propertySources": []
}

I checked also http://localhost:8888/cc-brain/default ... same

{
  "name": "cc-brain",
  "profiles": [
    "default"
  ],
  "label": null,
  "version": null,
  "state": null,
  "propertySources": []
}

but the configserver log looks okay to me ..

cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.482Z  INFO 1 --- [cc-configserver] [nio-8888-exec-8] .s.c.k.c.KubernetesEnvironmentRepository : Profiles: default                                                                                                              │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.482Z  INFO 1 --- [cc-configserver] [nio-8888-exec-8] .s.c.k.c.KubernetesEnvironmentRepository : Application: cc-brain                                                                                                          │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.482Z  INFO 1 --- [cc-configserver] [nio-8888-exec-8] .s.c.k.c.KubernetesEnvironmentRepository : Label: null                                                                                                                    │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.482Z DEBUG 1 --- [cc-configserver] [nio-8888-exec-8] .c.k.c.KubernetesConfigServerEnvironment : Activating profiles [default]                                                                                                  │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.483Z DEBUG 1 --- [cc-configserver] [nio-8888-exec-8] .c.k.c.c.KubernetesClientConfigMapsCache : Loaded (from cache) all config maps in namespace 'configcluster'                                                               │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.483Z DEBUG 1 --- [cc-configserver] [nio-8888-exec-8] .c.k.c.c.KubernetesClientConfigMapsCache : Loaded (from cache) all config maps in namespace 'configcluster'                                                               │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.483Z DEBUG 1 --- [cc-configserver] [nio-8888-exec-8] o.s.c.k.commons.config.ConfigUtils       : Found source with name : 'application in namespace: 'configcluster'                                                            │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:21:05.483Z DEBUG 1 --- [cc-configserver] [nio-8888-exec-8] o.s.c.k.c.c.SourceDataEntriesProcessor   : The single property with name: [application.yaml] will be treated as a yaml file                                               │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:55.320Z  INFO 1 --- [cc-configserver] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 30871562 and Watch connection expired: too old resource version: 30871562 (30873302) , will retry w/o resource │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:55.320Z  INFO 1 --- [cc-configserver] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon                                   │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.321Z  INFO 1 --- [cc-configserver] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...                                            │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.404Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap application was updated in namespace configcluster                                                                   │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.404Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.404Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap application-brain-client was updated in namespace configcluster                                                      │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.405Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.405Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap application-brain-member was updated in namespace configcluster                                                      │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.405Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.405Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap application-engines was updated in namespace configcluster                                                           │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap application-sba-client was updated in namespace configcluster                                                        │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap cc-configserver was updated in namespace configcluster                                                               │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace configcluster                                                              │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload                                                                             │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap logback-config was updated in namespace configcluster                                                                │
│ cc-configserver-deployment-6656b854f4-tq8vl 2024-08-13T18:22:56.406Z DEBUG 1 --- [cc-configserver] [pool-3-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload

ahoehma commented 2 months ago

Any hint? I'm pretty lost here :-)

ryanjbaxter commented 2 months ago

This is working fine for me

kubectl get configmaps
NAME                       DATA   AGE
frustration-free-k8s       1      3m56s
frustration-free-k8s-dev   1      4m54s
kube-root-ca.crt           1      18h
kubernetesconfigserver     1      4m54s
mysql                      1      4m48s

And if I make a request to config server at frustration-free-k8s/dev, I get properties sources from frustration-free-k8s and frustration-free-k8s-dev

{
   "name":"frustration-free-k8s",
   "profiles":[
      "dev"
   ],
   "label":null,
   "version":null,
   "state":null,
   "propertySources":[
      {
         "name":"configmap.frustration-free-k8s.frustration-free-k8s-dev.default.dev",
         "source":{
            "from-profile":"dev"
         }
      },
      {
         "name":"configmap.frustration-free-k8s.default.default",
         "source":{
            "spring.cloud.config.enabled":true,
            "eureka.client.enabled":false,
            "spring.cloud.kubernetes.discovery.discovery-server-url":"http://spring-cloud-kubernetes-discoveryserver",
            "spring.cloud.kubernetes.discovery.enabled":true
         }
      },
      {
         "name":"https://github.com/ryanjbaxter/s12024-config/application.yaml",
         "source":{
            "spring.sql.init.mode":"always",
            "spring.sql.init.continue-on-error":true,
            "management.endpoints.web.exposure.include":"*",
            "management.endpoint.env.show-values":"always"
         }
      }
   ]
}

ahoehma commented 2 months ago

@ryanjbaxter imagine you have this configmaps:

$k get cm -n configcluster
NAME                   DATA   AGE
application            1      20h
application-engines    1      20h
cc-brain-default       1      20h
cc-configserver        1      20h
cc-dm-default          1      20h
cc-sce-default         1      20h
kube-root-ca.crt       1      20h
logback-config         1      20h

(ignore the "-default" ... I'm playing here we a previous version - as far as I understand you "-default" is not necessary).

What I would like to do is this:

the service cc-brain should get the maps "application" and "cc-brain-default"
the service cc-sce should get the maps "application", "application-engines" and "cc-sce-default"
the service cc-dm should get the maps "application", "application-engines" and "cc-dm-default"
... and so on

How to enable this

on configserver side
on application side

ryanjbaxter commented 2 months ago

So you would need to have a profile enabled called engines for these two statements to work

the service cc-sce should get the maps "application", "application-engines" and "cc-sce-default" the service cc-dm should get the maps "application", "application-engines" and "cc-dm-default"

ahoehma commented 2 months ago

@ryanjbaxter where should I do this?

I tried to add this to the config-map 'cc-sce-default'

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: configcluster
  name: cc-sce # <app-name>-<profile> must be like this so that the configserver can find it
  labels:
    app.kubernetes.io/name: cc-sce
    app.kubernetes.io/component: sce-service
    app.kubernetes.io/part-of: configuration-cluster
    app.kubernetes.io/managed-by: kubectl
data:
  # only one key per ConfigMap - only then the ConfigServer can use it
  application.yaml: |
   .... my stuff here ....
    spring:
      profiles:
        include: 
        - engines   # will this include the configmap "application-engines"?

I tried also to run my cc-sce deployment with this env variable:

env:
        # application properties 
        - name: SPRING_APPLICATION_NAME
          value: "cc-sce"
        - name: SPRING_PROFILES_ACTIVE
          value: "engines"
        - name: SPRING_CONFIG_IMPORT
          value: "configserver:http://cc-configserver.configcluster:8888/"

Both are not really working for me.

What do I wrong? 😄

ryanjbaxter commented 2 months ago

Honestly you will need to provide an example showing this not working at this point. I am not sure why it doesnt work just by the info you provided

spring-cloud / spring-cloud-kubernetes

Spring Cloud Config Shared Configuration #136