Live update is not working with Spring Cloud Kubernetes Configuration Watcher

AngelJava78 commented 3 months ago

Live update is not working with Spring Cloud Kubernetes Configuration Watcher I would appreciate any feedback. I have a Spring Boot 3.3.0 project and Java 21. I would like to use the Spring Cloud Kubernetes Configuration Watcher feature in my demo project to update configuration from Kubernetes ConfigMaps or Secrets. I have tested my project with Spring Boot 2.X and Java 17 and it works fine, but not with Spring Boot 3.X. Here is my complete code. I have added spring-cloud-starter-kubernetes-client-config and spring-boot-starter-actuator to my pom.xml file.

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.3.0</version>
    <relativePath />
</parent>
<properties>
    <java.version>21</java.version>
    <maven.compiler.target>21</maven.compiler.target>
    <maven.compiler.source>21</maven.compiler.source>
    <spring-cloud.version>2023.0.2</spring-cloud.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-kubernetes-client-config</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-configuration-processor</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
</dependencies>

My java code:

@ConfigurationProperties("mydata")
@RefreshScope
@Component
public class MyDataProperties {
    private String name;

    private String email;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

}

@RestController
@RequestMapping
@Slf4j
@RefreshScope
public class DemoController {

    @Autowired
    private MyDataProperties myDataProperties;

    @GetMapping
    public ResponseEntity<String> sayHello() {
        String message = "Name: " + myDataProperties.getName() + ". Email: " + myDataProperties.getEmail();
        log.info("Response: {}", message);
        return new ResponseEntity<>(message, HttpStatus.OK);
    }

}

My Dockerfile

FROM openjdk:21-jdk
COPY target/demo-config-watcher-service.jar app.jar
ENTRYPOINT ["java", "-jar", "-Dspring.config.location=/etc/config/application.yml", "/app.jar", "--spring.profiles.active=${SPRING_PROFILES_ACTIVE}"]
EXPOSE 8080

Spring Cloud Kubernetes Configuration Watcher is running OK on Kubernetes, following step by step explained in Deployment.yaml

Logs

2024-06-12T18:24:34.796-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 182938 and Watch connection expired: too old resource version: 182938 (183367) , will retry w/o resourceVersion next time
2024-06-12T18:24:34.804-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Watch expired, will re-list-watch soon
2024-06-12T18:24:35.809-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...

Configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-configmap
  labels:
    spring.cloud.kubernetes.config: "true"
  annotations:
    spring.cloud.kubernetes.configmap.apps: "demo-config-watcher"
data:
  application.yml: |-
    management:
      endpoint:
        env:
          enabled: true
          show-values: ALWAYS
        refresh:
          enabled: true
        restart:
          enabled: true
      endpoints:
        web:
          exposure:
            include: '*'

    mydata:
      name: John Smith
      email: johnsmith@example.com

Deployment.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
     run: demo-config-watcher
  name: demo-config-watcher
  annotations:
    boot.spring.io/actuator: http://:8080/actuator
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 8080
  selector:
     run: demo-config-watcher
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-config-watcher
spec:
  selector:
    matchLabels:
      run: demo-config-watcher
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        run: demo-config-watcher
    spec:
      serviceAccountName: spring-cloud-kubernetes-configuration-watcher
      containers:
      - name: demo-config-watcher
        image: angeljava/demo-config-watcher:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "default"
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
        - name: config-volume
          configMap:
            name: demo-configmap

To start my app on Kubernetes

$ kubectl apply -f configmap.yaml
$ kubectl apply -f deployment.yaml

Modifiyng configmap to set new values for my data.

$ nano configmap.yaml
$ kubectl apply -f configmap.yaml

Logs after change and apply configmap.yaml

2024-06-12T17:42:51.704-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-12T17:42:51.705-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : will schedule remote refresh based on apps : [demo-config-watcher]
2024-06-12T17:42:51.706-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-12T17:42:51.706-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : Scheduling remote refresh event to be published for config-map: with appName : demo-config-watcher to be published in 10000 milliseconds
2024-06-12T17:42:51.706-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : onEvent config-map:
//Omitted for brevity
2024-06-12T17:43:01.738-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] o.s.c.k.c.d.DiscoveryClientUtils         : endpoint ports has a single entry, using port : 8080
2024-06-12T17:43:01.738-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Metadata actuator uri is: http://:8080/actuator
2024-06-12T17:43:01.739-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Found actuator URI in service instance metadata
2024-06-12T17:43:01.741-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Sending refresh request for demo-config-watcher to URI http://10.244.0.136:8080/actuator/refresh
2024-06-12T17:43:02.307-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-1] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.136:8080/actuator/refresh returned a 200 OK

In this step I continue to see the previous values, despite configuring the SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY environment variable.

env:
  - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CONFIGURATION_WATCHER
    value: DEBUG
  - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CLIENT_CONFIG_RELOAD
    value: DEBUG
  - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_COMMONS_CONFIG_RELOAD
    value: DEBUG
  - name: logging.level.org.springframework.cloud.kubernetes
    value: "DEBUG"
  - name: SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY
    value: "10000"

My demo app has actuator/refresh endpoint enabled, but it is not working fine.

$ minikube service list
$ curl http://{my-service-url}:{my-service-port}/actuator
Response:
{
    "_links": {
        //Endpoints were omitted for brevity
        "refresh": {
            "href": "http://192.168.49.2:32352/actuator/refresh",
            "templated": false
        },
        "restart": {
            "href": "http://192.168.49.2:32352/actuator/restart",
            "templated": false
        },
    }
}

If I connect to bash of my pod, I see application.yml updated with new values.

$ kubectl exec -it demo-config-watcher-f8bdf9c68-44ct4 -- sh
sh-4.4# cat /etc/config/application.yml

I would appreciate any kind of help.

Thanks in advance.

wind57 commented 3 months ago

I started to look at this and will let you know my findings.

wind57 commented 3 months ago

You are missing RBAC in your sample for spring-cloud-kubernetes-configuration-watcher. I've added the ones above from our integration tests:

cluster-role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-role
rules:
  - apiGroups: ["", "extensions", "apps", "discovery.k8s.io"]
    resources: ["configmaps", "pods", "services", "endpoints", "secrets", "endpointslices"]
    verbs: ["get", "list", "watch"]

cluster-role-binding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: demo-config-watcher
  name: demo-config-watcher
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: cluster-role
subjects:
  - kind: ServiceAccount
    name: spring-cloud-kubernetes-configuration-watcher
    namespace: default

service-account.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: demo-config-watcher
  name: spring-cloud-kubernetes-configuration-watcher

I had to change the parameter : imagePullPolicy: IfNotPresent, because I am testing this locally, with kind.

Your SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY is too small. I can re-produce the same behavior you see, if that value is set to 10000 (10 seconds), but if I go to a higher value, like 30000 (30 seconds), it works as expected.

AngelJava78 commented 3 months ago

@wind57 Thank you very much for your quick response.

I implemented the recommendations you gave me.

I changed the Role object to ClusterRole and the RoleBinding to ClusterRoleBinding.

You can see the changes in this folder of my Github repository.

I also updated the SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY variable with the value: 60000 (one minute).

However, the behavior seems a bit strange.

Here are some pieces of evidence.

I get all the pods.

$ kubectl get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
demo-config-watcher-f8bdf9c68-5686h                               1/1     Running   0          4s
demo-config-watcher-f8bdf9c68-vp2d6                               1/1     Running   0          4s
spring-cloud-kubernetes-configuration-watcher-deployment-7gp652   1/1     Running   0          51m

Here are some logs from the spring-cloud-kubernetes-configuration-watcher.

$ kubectl logs -f spring-cloud-kubernetes-configuration-watcher-deployment-7gp652
2024-06-13T05:38:20.690-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : No change detected in config maps/secrets, reload will not happen
2024-06-13T05:48:48.097-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 198347 and Watch connection expired: too old resource version: 198347 (198750) , will retry w/o resourceVersion next time
2024-06-13T05:48:48.098-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Watch expired, will re-list-watch soon
2024-06-13T05:48:49.099-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2024-06-13T05:53:22.397-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-13T05:53:22.399-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : onEvent config-map: class V1ConfigMap {
    apiVersion: v1
    binaryData: null
    data: {application.yml=management:
      endpoint:
        env:
          enabled: true
          show-values: ALWAYS
        refresh:
          enabled: true
        restart:
          enabled: true
      endpoints:
        web:
          exposure:
            include: '*'

    mydata:
      name: John Smith
      email: john.smith@gmail.com}
    immutable: null
    kind: ConfigMap
    metadata: class V1ObjectMeta {
        annotations: {kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"application.yml":"management:\n  endpoint:\n    env:\n      enabled: true\n      show-values: ALWAYS\n    refresh:\n      enabled: true\n    restart:\n      enabled: true\n  endpoints:\n    web:\n      exposure:\n        include: '*'\n\nmydata:\n  name: John Smith\n  email: john.smith@gmail.com"},"kind":"ConfigMap","metadata":{"annotations":{"spring.cloud.kubernetes.configmap.apps":"demo-config-watcher"},"labels":{"spring.cloud.kubernetes.config":"true"},"name":"demo-configmap","namespace":"default"}}
        , spring.cloud.kubernetes.configmap.apps=demo-config-watcher}
        creationTimestamp: 2024-06-13T10:49:58Z
        deletionGracePeriodSeconds: null
        deletionTimestamp: null
        finalizers: null
        generateName: null
        generation: null
        labels: {spring.cloud.kubernetes.config=true}
        managedFields: [class V1ManagedFieldsEntry {
            apiVersion: v1
            fieldsType: FieldsV1
            fieldsV1: {f:data={.={}, f:application.yml={}}, f:metadata={f:annotations={.={}, f:kubectl.kubernetes.io/last-applied-configuration={}, f:spring.cloud.kubernetes.configmap.apps={}}, f:labels={.={}, f:spring.cloud.kubernetes.config={}}}}
            manager: kubectl-client-side-apply
            operation: Update
            subresource: null
            time: 2024-06-13T11:53:22Z
        }]
        name: demo-configmap
        namespace: default
        ownerReferences: null
        resourceVersion: 199371
        selfLink: null
        uid: 94761a53-6e1e-44d8-a46f-39f0aaffb0c1
    }
}
2024-06-13T05:53:22.399-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.commons.config.ConfigUtils       : ConfigMap name has not been set, taking it from property/env spring.application.name (default=application)
2024-06-13T05:53:22.399-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-13T05:53:22.399-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : will schedule remote refresh based on apps : [demo-config-watcher]
2024-06-13T05:53:22.399-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : Scheduling remote refresh event to be published for config-map: with appName : demo-config-watcher to be published in 60000 milliseconds
2024-06-13T05:53:22.400-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] s.c.k.c.c.ConfigMapPropertySourceLocator : Config Map normalized sources : [{ config-map name : 'Optional[spring-cloud-kubernetes-configuration-watcher]', namespace : 'Optional.empty', prefix : '[ConfigUtils.Prefix@9bd0fa6 name = 'DEFAULT']' }]
2024-06-13T05:53:22.400-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.client.KubernetesClientUtils     : configmap namespace from provider : null
2024-06-13T05:53:22.413-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] .c.k.c.c.KubernetesClientConfigMapsCache : Loaded all config maps in namespace 'default'
2024-06-13T05:53:22.413-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] s.c.k.c.c.ConfigMapPropertySourceLocator : paths property sources : []
2024-06-13T05:53:22.414-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from locateMapPropertySources : ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-06-13T05:53:22.415-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : sources from locateMapPropertySources : [KubernetesClientConfigMapPropertySource@330445033 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]
2024-06-13T05:53:22.416-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from findPropertySources: ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-06-13T05:53:22.416-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment sources from findPropertySources : [MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]
2024-06-13T05:53:22.416-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : findPropertySources : [configmap.spring-cloud-kubernetes-configuration-watcher.default]
2024-06-13T05:53:22.416-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : no changes found, reload will not happen
2024-06-13T05:53:22.416-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : No change detected in config maps/secrets, reload will not happen
2024-06-13T05:54:22.401-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] s.c.k.c.d.KubernetesDiscoveryClientUtils : service labels from properties are empty, service with name : 'demo-config-watcher' will match
2024-06-13T05:54:22.402-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] o.s.c.k.c.d.DiscoveryClientUtils         : Adding labels metadata: {run=demo-config-watcher} for serviceId: demo-config-watcher
2024-06-13T05:54:22.402-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] o.s.c.k.c.d.DiscoveryClientUtils         : Adding annotations metadata: {boot.spring.io/actuator=http://:8080/actuator, kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"boot.spring.io/actuator":"http://:8080/actuator"},"labels":{"run":"demo-config-watcher"},"name":"demo-config-watcher","namespace":"default"},"spec":{"ports":[{"name":"http","port":8080,"targetPort":8080}],"selector":{"run":"demo-config-watcher"},"type":"LoadBalancer"}}
} for serviceId: demo-config-watcher
2024-06-13T05:54:22.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] o.s.c.k.c.d.DiscoveryClientUtils         : Adding port metadata: {port.http=8080} for serviceId : demo-config-watcher
2024-06-13T05:54:22.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] o.s.c.k.c.d.DiscoveryClientUtils         : endpoint ports has a single entry, using port : 8080
2024-06-13T05:54:22.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Metadata actuator uri is: http://:8080/actuator
2024-06-13T05:54:22.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Found actuator URI in service instance metadata
2024-06-13T05:54:22.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Sending refresh request for demo-config-watcher to URI http://10.244.0.153:8080/actuator/refresh
2024-06-13T05:54:22.409-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Metadata actuator uri is: http://:8080/actuator
2024-06-13T05:54:22.410-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Found actuator URI in service instance metadata
2024-06-13T05:54:22.411-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-2] tesClientEventBasedSecretsChangeDetector : Sending refresh request for demo-config-watcher to URI http://10.244.0.154:8080/actuator/refresh
2024-06-13T05:54:22.894-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-3] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.153:8080/actuator/refresh returned a 200 OK
2024-06-13T05:54:22.918-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-4] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.154:8080/actuator/refresh returned a 200 OK
2024-06-13T05:54:44.128-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 199013 and Watch connection expired: too old resource version: 199013 (199215) , will retry w/o resourceVersion next time
2024-06-13T05:54:44.129-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Watch expired, will re-list-watch soon
2024-06-13T05:54:45.131-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2024-06-13T05:56:02.403-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-13T05:56:02.404-06:00  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : will schedule remote refresh based on apps : [demo-config-watcher]
2024-06-13T05:56:02.404-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap demo-configmap was updated in namespace default
2024-06-13T05:56:02.405-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : onEvent config-map: class V1ConfigMap {
    apiVersion: v1
    binaryData: null
    data: {application.yml=management:
      endpoint:
        env:
          enabled: true
          show-values: ALWAYS
        refresh:
          enabled: true
        restart:
          enabled: true
      endpoints:
        web:
          exposure:
            include: '*'

    mydata:
      name: Sarah Johnson
      email: sarah.johnson@yahoo.com}
    immutable: null
    kind: ConfigMap
    metadata: class V1ObjectMeta {
        annotations: {kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"application.yml":"management:\n  endpoint:\n    env:\n      enabled: true\n      show-values: ALWAYS\n    refresh:\n      enabled: true\n    restart:\n      enabled: true\n  endpoints:\n    web:\n      exposure:\n        include: '*'\n\nmydata:\n  name: Sarah Johnson\n  email: sarah.johnson@yahoo.com"},"kind":"ConfigMap","metadata":{"annotations":{"spring.cloud.kubernetes.configmap.apps":"demo-config-watcher"},"labels":{"spring.cloud.kubernetes.config":"true"},"name":"demo-configmap","namespace":"default"}}
        , spring.cloud.kubernetes.configmap.apps=demo-config-watcher}
        creationTimestamp: 2024-06-13T10:49:58Z
        deletionGracePeriodSeconds: null
        deletionTimestamp: null
        finalizers: null
        generateName: null
        generation: null
        labels: {spring.cloud.kubernetes.config=true}
        managedFields: [class V1ManagedFieldsEntry {
            apiVersion: v1
            fieldsType: FieldsV1
            fieldsV1: {f:data={.={}, f:application.yml={}}, f:metadata={f:annotations={.={}, f:kubectl.kubernetes.io/last-applied-configuration={}, f:spring.cloud.kubernetes.configmap.apps={}}, f:labels={.={}, f:spring.cloud.kubernetes.config={}}}}
            manager: kubectl-client-side-apply
            operation: Update
            subresource: null
            time: 2024-06-13T11:56:02Z
        }]
        name: demo-configmap
        namespace: default
        ownerReferences: null
        resourceVersion: 199577
        selfLink: null
        uid: 94761a53-6e1e-44d8-a46f-39f0aaffb0c1
    }
}
2024-06-13T05:56:02.405-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.commons.config.ConfigUtils       : ConfigMap name has not been set, taking it from property/env spring.application.name (default=application)
2024-06-13T05:56:02.405-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.watcher.WatcherUtil            : Scheduling remote refresh event to be published for config-map: with appName : demo-config-watcher to be published in 60000 milliseconds
2024-06-13T05:56:02.407-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] s.c.k.c.c.ConfigMapPropertySourceLocator : Config Map normalized sources : [{ config-map name : 'Optional[spring-cloud-kubernetes-configuration-watcher]', namespace : 'Optional.empty', prefix : '[ConfigUtils.Prefix@9bd0fa6 name = 'DEFAULT']' }]
2024-06-13T05:56:02.407-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.client.KubernetesClientUtils     : configmap namespace from provider : null
2024-06-13T05:56:02.420-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] .c.k.c.c.KubernetesClientConfigMapsCache : Loaded all config maps in namespace 'default'
2024-06-13T05:56:02.420-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] s.c.k.c.c.ConfigMapPropertySourceLocator : paths property sources : []
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from locateMapPropertySources : ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : sources from locateMapPropertySources : [KubernetesClientConfigMapPropertySource@1884753967 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from findPropertySources: ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment sources from findPropertySources : [MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource@1229790531 {name='configmap.spring-cloud-kubernetes-configuration-watcher.default', properties={}}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : findPropertySources : [configmap.spring-cloud-kubernetes-configuration-watcher.default]
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : no changes found, reload will not happen
2024-06-13T05:56:02.421-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-11-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : No change detected in config maps/secrets, reload will not happen
2024-06-13T05:57:02.407-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] s.c.k.c.d.KubernetesDiscoveryClientUtils : service labels from properties are empty, service with name : 'demo-config-watcher' will match
2024-06-13T05:57:02.407-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] o.s.c.k.c.d.DiscoveryClientUtils         : Adding labels metadata: {run=demo-config-watcher} for serviceId: demo-config-watcher
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] o.s.c.k.c.d.DiscoveryClientUtils         : Adding annotations metadata: {boot.spring.io/actuator=http://:8080/actuator, kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"boot.spring.io/actuator":"http://:8080/actuator"},"labels":{"run":"demo-config-watcher"},"name":"demo-config-watcher","namespace":"default"},"spec":{"ports":[{"name":"http","port":8080,"targetPort":8080}],"selector":{"run":"demo-config-watcher"},"type":"LoadBalancer"}}
} for serviceId: demo-config-watcher
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] o.s.c.k.c.d.DiscoveryClientUtils         : Adding port metadata: {port.http=8080} for serviceId : demo-config-watcher
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] o.s.c.k.c.d.DiscoveryClientUtils         : endpoint ports has a single entry, using port : 8080
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Metadata actuator uri is: http://:8080/actuator
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Found actuator URI in service instance metadata
2024-06-13T05:57:02.408-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Sending refresh request for demo-config-watcher to URI http://10.244.0.153:8080/actuator/refresh
2024-06-13T05:57:02.417-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Metadata actuator uri is: http://:8080/actuator
2024-06-13T05:57:02.419-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Found actuator URI in service instance metadata
2024-06-13T05:57:02.419-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [oundedElastic-1] tesClientEventBasedSecretsChangeDetector : Sending refresh request for demo-config-watcher to URI http://10.244.0.154:8080/actuator/refresh
2024-06-13T05:57:02.519-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-1] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.153:8080/actuator/refresh returned a 200 OK
2024-06-13T05:57:02.661-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-2] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.154:8080/actuator/refresh returned a 200 OK

Part of the logs from the first pod.

$ kubectl logs -f demo-config-watcher-f8bdf9c68-5686h
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.3.0)

2024-06-13T05:40:18.134-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : Starting DemoConfigWatcherServiceApplication v0.0.1-SNAPSHOT using Java 21 with PID 1 (/app.jar started by root in /)
2024-06-13T05:40:18.146-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : The following 2 profiles are active: "kubernetes", "default"
2024-06-13T05:40:22.029-06:00  INFO 1 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=36f7a0b0-ac51-3377-8116-7fbb02e14332
2024-06-13T05:40:23.261-06:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-06-13T05:40:23.287-06:00  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-06-13T05:40:23.288-06:00  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.24]
2024-06-13T05:40:23.352-06:00  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-06-13T05:40:23.355-06:00  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4941 ms
2024-06-13T05:40:26.309-06:00  INFO 1 --- [           main] o.s.c.k.client.KubernetesClientUtils     : Created API client in the cluster.
2024-06-13T05:40:27.253-06:00  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 19 endpoints beneath base path '/actuator'
2024-06-13T05:40:27.401-06:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2024-06-13T05:40:27.459-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : Started DemoConfigWatcherServiceApplication in 11.59 seconds (process running for 13.284)
2024-06-13T05:44:55.435-06:00  INFO 1 --- [nio-8080-exec-2] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-06-13T05:44:55.435-06:00  INFO 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-06-13T05:44:55.439-06:00  INFO 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : Completed initialization in 4 ms
2024-06-13T05:54:29.053-06:00  INFO 1 --- [nio-8080-exec-7] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com
2024-06-13T05:56:09.940-06:00  INFO 1 --- [nio-8080-exec-6] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com
2024-06-13T05:57:13.411-06:00  INFO 1 --- [nio-8080-exec-9] c.a.services.controller.DemoController   : Response: Name: Sarah Johnson. Email: sarah.johnson@yahoo.com

Part of the logs from the second pod.

$ kubectl logs -f demo-config-watcher-f8bdf9c68-vp2d6
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.3.0)

2024-06-13T05:40:18.026-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : Starting DemoConfigWatcherServiceApplication v0.0.1-SNAPSHOT using Java 21 with PID 1 (/app.jar started by root in /)
2024-06-13T05:40:18.101-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : The following 2 profiles are active: "kubernetes", "default"
2024-06-13T05:40:22.197-06:00  INFO 1 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=36f7a0b0-ac51-3377-8116-7fbb02e14332
2024-06-13T05:40:23.262-06:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-06-13T05:40:23.293-06:00  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-06-13T05:40:23.295-06:00  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.24]
2024-06-13T05:40:23.356-06:00  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-06-13T05:40:23.358-06:00  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4884 ms
2024-06-13T05:40:26.184-06:00  INFO 1 --- [           main] o.s.c.k.client.KubernetesClientUtils     : Created API client in the cluster.
2024-06-13T05:40:27.263-06:00  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 19 endpoints beneath base path '/actuator'
2024-06-13T05:40:27.388-06:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2024-06-13T05:40:27.461-06:00  INFO 1 --- [           main] .a.s.DemoConfigWatcherServiceApplication : Started DemoConfigWatcherServiceApplication in 11.297 seconds (process running for 12.458)
2024-06-13T05:43:04.912-06:00  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-06-13T05:43:04.912-06:00  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-06-13T05:43:04.922-06:00  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 9 ms
2024-06-13T05:54:32.517-06:00  INFO 1 --- [nio-8080-exec-7] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com
2024-06-13T05:56:13.316-06:00  INFO 1 --- [io-8080-exec-10] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com
2024-06-13T05:57:16.677-06:00  INFO 1 --- [nio-8080-exec-1] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com
2024-06-13T06:01:08.338-06:00  INFO 1 --- [nio-8080-exec-3] c.a.services.controller.DemoController   : Response: Name: John Smith. Email: john.smith@gmail.com

As you can see, I updated the configmap at 2024-06-13T05:53:22.399-06:00 with these values:

    mydata:
      name: John Smith
      email: john.smith@gmail.com

Both pods were updated correctly.

Then, I updated the configmap at 2024-06-13T05:56:02.405-06:00 with these values.

    mydata:
      name: Sarah Johnson
      email: sarah.johnson@yahoo.com

Although spring-cloud-kubernetes-configuration-watcher ran the /actuator/refresh post, only one of the pods was updated correctly.

2024-06-13T05:57:02.519-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-1] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.153:8080/actuator/refresh returned a 200 OK
2024-06-13T05:57:02.661-06:00 DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [or-http-epoll-2] tesClientEventBasedSecretsChangeDetector : Refresh sent to demo-config-watcher at URI address http://10.244.0.154:8080/actuator/refresh returned a 200 OK

To make sure that each pod received an HTTP request, I connected to each pod and executed next commands:

$ kubectl exec -it demo-config-watcher-f8bdf9c68-5686h -- sh
sh-4.4# curl http://localhost:8080
Name: John Smith. Email: john.smith@gmail.com
sh-4.4# curl http://localhost:8080
Name: Sarah Johnson. Email: sarah.johnson@yahoo.com
sh-4.4#

$ kubectl exec -it demo-config-watcher-f8bdf9c68-vp2d6 -- sh
sh-4.4# curl http://localhost:8080
Name: John Smith. Email: john.smith@gmail.com
sh-4.4# curl http://localhost:8080
Name: John Smith. Email: john.smith@gmail.com
sh-4.4#

I would appreciate your help in reviewing what I am doing wrong, please.

Thanks a lot!

wind57 commented 3 months ago

well... this is rather interesting. I can re-produce your issue once I go to 2 replicas indeed.

Now why that happens, I don't know, but I will debug the issue and try to figure it out.

I'll update this post once I have the findings.

wind57 commented 3 months ago

I've re-produced this one once, but can't do it anymore and I have tried a few times already. weird

wind57 commented 3 months ago

@ryanjbaxter this is a bug, I don't know where exactly, but it is. If I increase the replicas to 5 it is easy to reproduce . I'll try to figure out what is going on. Would you please add the label?

wind57 commented 3 months ago

investigation update:

Initially I thought that there is something within our code base and we have a bug somewhere. To confirm (or infirm) this, I've added such a class to your sample:

package com.angeljava.services.event;

import org.springframework.cloud.context.scope.refresh.RefreshScopeRefreshedEvent;
import org.springframework.context.ApplicationListener;
import org.springframework.stereotype.Component;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

@Component
public class EventCatcher implements ApplicationListener<RefreshScopeRefreshedEvent> {

    private final AtomicInteger times = new AtomicInteger(0);

    @Override
    public void onApplicationEvent(RefreshScopeRefreshedEvent event) {
        times.incrementAndGet();
        List<String> allLines;
        try {
            allLines = Files.readAllLines(new File("/etc/config/application.yml").toPath());
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        System.out.println("caught event : " + event.toString() + " from source : " + event.getSource()
            + " at " + times.get());
        System.out.println("all lines : " + allLines);
    }

    public int getTimes() {
        return times.get();
    }
}

I also add a controller method:

    @GetMapping("/times")
    public ResponseEntity<Integer> times() {
        return new ResponseEntity<>(eventCatcher.getTimes(), HttpStatus.OK);
    }

To reproduce the issue I've added 5 replicas in the deployment also. So here is what I do, step by step:

I start with:

    mydata:
      name: ER
      email: johnsmith@example.com

in the configmap; deploy the app and go to a certain pod in my kind cluster.

execute a curl:

root@kind-control-plane:/# curl 10.244.0.5:8080
Name: ER. Email: johnsmith@example.comroot@kind-control-plane:/#

and:

root@kind-control-plane:/# curl 10.244.0.5:8080/times
1

OK, so that tells me that configuration watcher and app are correctly working.

Now, I change the value in the configmap:

    mydata:
      name: AT
      email: johnsmith@example.com

wait 1 minute and go to the pods. Some pods are working as expected, but this one:

root@kind-control-plane:/# curl 10.244.0.5:8080/times
2

kubectl logs demo-config-watcher-6694f7cc74-n5hqs

caught event :  ....... mydata:,   name: ER,   email: johnsmith@example.com]

This tells me two things, first is that the refresh event was received (meaning that our code does not have problems with firing the events). The second thing I notice, is that at the time when the refresh event was received, the value in the mount was ER. This is inconsistent across pods, because other pods, when the refresh event was received see the updated value as AT.

As a last step, after a few minutes, if I do :

wind57@wind57s-MacBook-Pro /p/t/kubernetes-configuration-watcher (main)> kubectl exec -it demo-config-watcher-6694f7cc74-n5hqs -- cat /etc/config/application.yml | grep mydata -A 5
mydata:
  name: AT
  email: johnsmith@example.com

I see that the value is finally updated to AT. But at this point, it does not even matter anymore, as the refresh event was fired when the value in the mount was ER...

In the end, this is not our code-base issue (although I wish it were, cause in that case I could have tried to fix it). This looks like kubernetes eventual consistency problem, also looks like it is "aggravated" when there are more replicas. I will try to ask on the k8s github or figure out something, but in the end, I highly doubt that anything is really possible in your case. The only hope you have is to increase that value to more the 60 seconds...

If you abstract yourself away for a second, this issue is a little frustrating for a different reason too. It is related to this one.

Think about it, if you are mounting a file, why would you need configuration watcher at all (with all of its RBAC rules)? Why can't we have a mechanism in place to watch only a file/folder? We would not need to talk to the kube-api at all :) When we see that it changes, refresh the app. I plan to work on such a proposal next year, after we drop the "path support" (this PR)

AngelJava78 commented 3 months ago

@wind57 Thanks a lot!!! I'm going to set SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY variable to 5 minutes and test my app. I'm hopeful that this will resolve the issue and I will post an update with my findings later on

AngelJava78 commented 3 months ago

Thank you so much @wind57 for your valuable advice and suggestions.

I'm happy to report that my service and Spring Cloud Kubernetes Configuration Watcher are working perfectly fine now. I followed your advice and set up SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY variable to 2 minutes and also increased the number of replicas to 10, which has improved the performance significantly.

However, I'm facing some issues while trying to implement the same with Secrets. I have updated my code in my Github repository, and I was hoping if you could kindly review it and provide me with feedback on what I'm missing or doing wrong.

Once again, I'm deeply grateful for your help, and I look forward to hearing from you soon.

wind57 commented 3 months ago

I will take a look, of course, and update this post once done. Im not sure I can do it this week, since Im on vacation, but will try.

wind57 commented 3 months ago

It would be great if what is not working would not be this generic:

However, I'm facing some issues while trying to implement the same with Secrets

I mean: what issues exactly your are facing, because otherwise I have to guess. Let's see if I guessed correctly...

I changed imagePullPolicy: IfNotPresent, not sure why you have it like this if you are testing locally with minukube. Just a minor note.

Then, I renamed some files from Kubernetes directory, so that there is an order there for sure when deploying:

0_secret.yaml
1_configmap.yaml
2_deployment.yaml

I see that you are using : "-Dspring.cloud.kubernetes.secrets.paths=/etc/secrets/db-secret". This is what we call "path support" and it is deprecated and will be removed in the next major release. Because of that, I would suggest we do not even debug the cause of your issue, but instead focus on the new (and supported in the future) functionality: spring.config.import.

You need to change your app in minimal ways:

drop from Dockerfile

"-Dspring.cloud.kubernetes.secrets.paths=/etc/secrets/db-secret",

and place this in application.yaml:

spring:
  config:
    import: "configtree:/etc/secrets/db-secret/"

If I start the app this way, I see this:

2024-06-19T01:41:06.343-06:00  INFO 1 --- [   scheduling-1] c.angeljava.services.scheduler.DemoBean  : From secret [db-secret]: Username: null. Password: null

Then I realized that you are using: "-Dspring.config.location=/etc/config/application.yml", which means that on deployment, this yaml is going to be used, not the one from src/main/resources. As such, I changed your 1_configmap.yaml also:

....

....    

spring:
      cloud:
        kubernetes:
          reload:
            enabled: true
          secrets:
            paths: /etc/secrets/db-secret

      config:
        import: "configtree:/etc/secrets/db-secret/"

(be careful to respect all indention in this file). Then I deploy the app and see in logs:

2024-06-19T01:54:31.870-06:00  INFO 1 --- [   scheduling-1] c.angeljava.services.scheduler.DemoBean  : From configmap [demo-configmap]: Name: William Clark. Email: williamc@yahoo.com
2024-06-19T01:54:31.871-06:00  INFO 1 --- [   scheduling-1] c.angeljava.services.scheduler.DemoBean  : From secret [db-secret]: Username: zuli@gmail.com. Password: May31th2024

So my guess was that you were no able to see those properties correctly wired to begin with... After you make these changes, configuration watcher should work the same as it did before

AngelJava78 commented 3 months ago

@wind57 I want to express my sincere gratitude for taking the time to review my code and providing me with an excellent solution to the bug I was facing. Your expertise and guidance have been invaluable, and I am truly grateful for your help.

Thank you once again for your generous support.

wind57 commented 3 months ago

@ryanjbaxter it seems like this one can be closed

wind57 commented 3 months ago

by the way, this was not a bug, may be worth to remove the label too

spring-cloud / spring-cloud-kubernetes

Live update is not working with Spring Cloud Kubernetes Configuration Watcher #1664