In rare scenarios ConfigMap update is not updated using polling strategy

[gmrammohan15]

Describe the bug Intermittently once pod is deployed, Any configMap resource update is not getting applied to the Application We see in problematic pods, we continously see below WARN messages 10:12:24.807 WARN --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place

This does not happen normally, but happens sometimes after which its not recoverable, we need to go for full deployment to overcome.No new configMap updates will be applied.

We see that ConfigMap resource is updated in K8s. we could verify using kubectl describe cm But K8s spring cloud is not able to accept or update the newly updated resource.

Version Details [INFO] +- org.springframework.cloud:spring-cloud-kubernetes-commons:jar:3.0.3:compile [INFO] | +- org.springframework.boot:spring-boot-autoconfigure:jar:3.1.4:compile [INFO] | +- org.springframework.cloud:spring-cloud-commons:jar:4.0.3:compile [INFO] | | - org.springframework.security:spring-security-crypto:jar:6.1.4:compile [INFO] | - org.springframework.cloud:spring-cloud-context:jar:4.0.3:compile [INFO] +- org.springframework.cloud:spring-cloud-kubernetes-fabric8-autoconfig:jar:3.0.3:compile [INFO] +- com.ciscospark:cisco-spark-starter-server:jar:4.1.8:compile [INFO] | +- com.ciscospark:cisco-spark-autoconfigure:jar:4.1.8:compile

[INFO] +- io.fabric8:kubernetes-client:jar:6.2.0:compile [INFO] | +- io.fabric8:kubernetes-client-api:jar:6.2.0:compile [INFO] | | +- io.fabric8:kubernetes-model-core:jar:6.2.0:compile [INFO] | | | - io.fabric8:kubernetes-model-common:jar:6.2.0:compile

Temp workaround Delete the pod, or go for full deployment(image, configmap)

logs snippet when issue happpened

10:12:24.806 WARN --- [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : path support is deprecated and will be removed in a future release. Please use spring.config.import 10:12:24.806 DEBUG --- [TaskScheduler-1] s.c.k.c.c.ConfigMapPropertySourceLocator : paths property sources : [] 10:12:24.806 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva'}, BootstrapPropertySource {name='bootstrapProperties-configmap.dhruva-proxy-static-config-map.wxc-dhruva'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='filter-properties.yaml'}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} 10:12:24.806 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@232678616 {name='configmap..wxc-dhruva', properties={}}] 10:12:24.806 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : left size: 1 10:12:24.806 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@232678616 {name='configmap..wxc-dhruva', properties={}} 10:12:24.806 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : right size: 2 10:12:24.807 DEBUG --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : Fabric8ConfigMapPropertySource@794279295 {name='configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva', properties={

MASKED### SECURITY REASONS

10:12:24.807 WARN --- [TaskScheduler-1] o.s.c.k.c.c.reload.ConfigReloadUtil : The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place

Sample app config

spring:
  application:
    name: dhruvaProxyApplication
  cloud:
    consul:
      enabled: false
    kubernetes:
      config:
        include-profile-specific-sources: false
      reload:
        enabled: true
        mode: polling
        period: 10000 # event based missed some config changes, testing polling
      discovery:
        enabled: false

JAVA_OPTS -Dspring.cloud.kubernetes.config.namespace=wxc-dhruva -Dspring.cloud.kubernetes.config.sources[0].name=dhruva-proxy-static-config-map -Dspring.cloud.kubernetes.config.sources[1].name=dhruva-proxy-dynamic-config-map

[wind57]

Something is very off here, tbh. The thing that bothers me the most from glancing over the logs is this:

[Fabric8ConfigMapPropertySource@232678616 {name='configmap..wxc-dhruva', properties={}}]

This name: configmap..wxc-dhruva is kind of impossible :| We compute the name of a source from (configmap/secrets) + + , so this one should really be:

configmap.dhruva-proxy-static-config-map.wxc-dhruva or configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva.

It has been a while since I've looked at that code, but I will look a bit more and try to figure out what is going on.

The second issue that bothers me too, is those logs left size: 1 and right size: 2, which implies something wrong again...

I will try to refresh my memory about things and will get back as soon as I have any hints of what is going on.

[wind57]

Is there a way you can provide debug logs from the entire org.springframework.cloud.kubernetes package, btw?

[gmrammohan15]

We had to restart the pod to get things working again, hence lost the logs.We have enabled debug logs in integration environment, will post if the issue gets reproduced again.

[gmrammohan15]

Just thinking , since most of the times it works fine, is there any corner scenario (e.g temp network issue or something like that) that gets into this state. Like when this happens, it never comes out of this state ("does not match the ones loaded from Kubernetes").However the new pods or other pods in the same cluster talking to the same API server works fine. Not sure when it gets in to this state if it always returns back from cache for every polling interval since actual call should have returned correct data. By default what is the cache expiry time.

[wind57]

We do have a cache in place, but it works differently, it is request based, not time based. But it's irrelevant for now, I will leave it aside for the time being.

What I can't get my head around is how this property source : configmap..wxc-dhruva came to life. The thing that I don't understand is that our code does something like this:

 1) what do I currently have in my environment?
 2) let me re-compute property sources based on your provided input
 3) compare (1) to (2) 

The weird part is that the answer to (1) (right size: 2) seems to be (from your trunked logs):

configmap.dhruva-proxy-static-config-map.wxc-dhruva
configmap.dhruva-proxy-dynamic-config-map.wxc-dhruva

Which means that we did manage to read correctly at the bootstrap time (your "app config" is really a bootstrap.yaml looks like), the properties which you have defined via JAVA_OPTS.

In other words you told us the namespace you use: wxc-dhruva, the two config maps we are supposed to read from : dhruva-proxy-static-config-map and dhruva-proxy-dynamic-config-map and we did that; that right size: 2 and those two BootstrapPropertySources, is proof that it worked.

When pooling happens, we go to environment, see that we have already those two loaded, and try to compute them again, from the same properties that you provided initially. At this point, from your logs, I see a property source with name configmap..wxc-dhruva which makes no sense at all. The name of the configmap is missing entirely, so I have no idea how we got to this point, honestly.

The debug logs I hope that you will provide, will bring me a bit closer to solving this puzzle.

[gmrammohan15]

Sure, will update once the issue is reproduced with logs.Will monitor and provide more details.

[wind57]

@ryanjbaxter can you please add the feedback tag please? It has been a while now...

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[gmrammohan15]

Issue got reproduced. Below are the sequence of logs.(Have masked few things with ### , we can consider that as some valid string)

org.springframework.cloud.kubernetes.commons.config.ConfigUtils

Operation: [list] for kind: [ConfigMap] with name: [null] in namespace: [###] failed.. Ignoring.

stack trace io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list] for kind: [ConfigMap] with name: [null] in namespace: [wxc-dhruva] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:452) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:418) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:97) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapsCache.lambda$byNamespace$0(Fabric8ConfigMapsCache.java:56) at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapsCache.byNamespace(Fabric8ConfigMapsCache.java:54) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils.strippedConfigMaps(Fabric8ConfigUtils.java:135) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils.configMapsDataByName(Fabric8ConfigUtils.java:127) at org.springframework.cloud.kubernetes.fabric8.config.NamedConfigMapContextToSourceDataProvider$1.dataSupplier(NamedConfigMapContextToSourceDataProvider.java:54) at org.springframework.cloud.kubernetes.commons.config.NamedSourceData.compute(NamedSourceData.java:51) at org.springframework.cloud.kubernetes.fabric8.config.NamedConfigMapContextToSourceDataProvider.lambda$get$0(NamedConfigMapContextToSourceDataProvider.java:57) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.lambda$getSourceData$0(Fabric8ConfigMapPropertySource.java:51) at java.base/java.util.Optional.map(Optional.java:260) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.getSourceData(Fabric8ConfigMapPropertySource.java:51) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.(Fabric8ConfigMapPropertySource.java:46) at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySourceLocator.getMapPropertySource(Fabric8ConfigMapPropertySourceLocator.java:61) at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:84) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:84) at org.springframework.cloud.kubernetes.commons.config.reload.ConfigReloadUtil.locateMapPropertySources(ConfigReloadUtil.java:121) at org.springframework.cloud.kubernetes.commons.config.reload.PollingConfigMapChangeDetector.executeCycle(PollingConfigMapChangeDetector.java:86) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.io.IOException: timeout at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:450) ... 28 common frames omitted Caused by: java.io.InterruptedIOException: timeout at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398) at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360) at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:209) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ... 3 common frames omitted Caused by: java.io.IOException: Canceled at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ... 4 common frames omitted

org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator paths property sources : []

sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@1301577353 {name='configmap..###', properties={}}]

left size: 1

Fabric8ConfigMapPropertySource@1301577353 {name='configmap..###', properties={}}

right size: 2

The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place

Once this exception is hit and there is mismatch in left size and right size, we don't recover even if is there is configMap update , it does not reload.

 

[ryanjbaxter]

Are these the debug logs that @wind57 requested as well?

[gmrammohan15]

yes, correct.

[wind57]

as usual, time is the limiting factor for me... I will be looking at this one, but no timeline when exactly.

[wind57]

The error happens in the KubernetesClient itself that we are using (fabric8) and it fails within its http library that it uses: okhttp3. There was a timeout when we tried to get the configmaps using that client, and it seems that the connection is canceled at that point in time. It goes something like this:

• give me the configmaps
• for that, fabric8 issues a http request (using okhttp3) to the k8s api server
• there was a timeout that happened when waiting for the response from above
• the library kills the request and throws a timeout exception

At least this is my understanding of the stacktrace above. The thing is, all of this happens outside our code in spring...

But either way, these are not full debug logs, I was expecting a lot more output and some debug statements that are not provided. The are many other log statements that we have in our code base that you have not provided, without them I can't help you

[gmrammohan15]

@wind57

Thanks for looking into it.Below are the logs that we could capture apart from the stack trace. Have highlighted certain section of logs.After the operation exception happens , it detects a change which is strange and should not happen.Post that subsequent polling always has left size and right size mismatch ( due to entry ..) and any change is configMap does not trigger refresh.

Note: Have masked some actual content of configMap due to security reasons.

2024-10-08 05:15:40.122Z Polling for changes in config maps - - - - -

2024-10-08 05:15:40.122Z environment sources from findPropertySources : [ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}] - - - - -

2024-10-08 05:15:40.122Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:15:40.122Z Config Map normalized sources : [{ config-map name : 'Optional[wxc-dhruva-proxy-static-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }, { config-map name : 'Optional[wxc-dhruva-proxy-dynamic-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }] - - - - -

2024-10-08 05:15:40.122Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:15:40.122Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:15:40.122Z findPropertySources : [configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno, configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno] - - - - -

2024-10-08 05:15:40.122Z environment from findPropertySources: ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:15:40.136Z The single property with name: [dynamic.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:15:40.136Z Found source with name : 'wxc-dhruva-proxy-dynamic-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:40.136Z Loaded all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:40.136Z Found source with name : 'wxc-dhruva-proxy-static-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:40.136Z Loaded (from cache) all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:40.136Z The single property with name: [static.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:15:40.136Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:15:40.142Z paths property sources : [] - - - - -

2024-10-08 05:15:40.142Z environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:15:40.143Z sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@218418285 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.#######(MASKED)... - - - - -

2024-10-08 05:15:40.143Z no changes found, reload will not happen - - - - -

2024-10-08 05:15:50.143Z Polling for changes in config maps - - - - -

2024-10-08 05:15:50.144Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:15:50.144Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:15:50.144Z environment sources from findPropertySources : [ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}] - - - - -

2024-10-08 05:15:50.144Z Config Map normalized sources : [{ config-map name : 'Optional[wxc-dhruva-proxy-static-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }, { config-map name : 'Optional[wxc-dhruva-proxy-dynamic-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }] - - - - -

2024-10-08 05:15:50.144Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:15:50.144Z environment from findPropertySources: ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:15:50.144Z findPropertySources : [configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno, configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno] - - - - -

2024-10-08 05:15:50.165Z The single property with name: [static.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:15:50.165Z Found source with name : 'wxc-dhruva-proxy-static-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:50.165Z Loaded all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:50.166Z Found source with name : 'wxc-dhruva-proxy-dynamic-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:50.166Z The single property with name: [dynamic.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:15:50.166Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:15:50.166Z Loaded (from cache) all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:15:50.171Z paths property sources : [] - - - - -

2024-10-08 05:15:50.171Z environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:15:50.172Z no changes found, reload will not happen - - - - -

2024-10-08 05:15:50.172Z sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@1929727208 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.#####(MASKED).... - - - - -

2024-10-08 05:16:00.172Z environment sources from findPropertySources : [ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}] - - - - -

2024-10-08 05:16:00.172Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:16:00.172Z environment from findPropertySources: ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:16:00.172Z Config Map normalized sources : [{ config-map name : 'Optional[wxc-dhruva-proxy-static-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }, { config-map name : 'Optional[wxc-dhruva-proxy-dynamic-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }] - - - - -

2024-10-08 05:16:00.172Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:16:00.172Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:00.172Z findPropertySources : [configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno, configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno] - - - - -

2024-10-08 05:16:00.172Z Polling for changes in config maps - - - - -

2024-10-08 05:16:10.175Z Operation: [list] for kind: [ConfigMap] with name: [null] in namespace: [wxc-dhruva-mno] failed.. Ignoring. - - - - -

2024-10-08 05:16:10.176Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:11.990Z Loaded all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:11.990Z The single property with name: [dynamic.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:16:11.990Z Found source with name : 'wxc-dhruva-proxy-dynamic-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:11.996Z environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:16:11.996Z paths property sources : [] - - - - -

2024-10-08 05:16:11.997Z Detected change in config maps - - - - -

2024-10-08 05:16:11.997Z sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@1567844113 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.#####MASKED.... - - - - -

2024-10-08 05:16:11.997Z found change in : Fabric8ConfigMapPropertySource@318976780 {name='configmap..wxc-dhruva-mno', properties={}} - - - - -

2024-10-08 05:16:11.998Z Reloading using strategy: REFRESH - - - - -

2024-10-08 05:16:12.102Z 'kubernetes' already in list of active profiles - - - - -

2024-10-08 05:16:12.102Z Adding 'kubernetes' to list of active profiles - - - - -

2024-10-08 05:16:12.174Z Created API client in the cluster. - - - - -

2024-10-08 05:16:12.175Z Service account namespace value: /var/run/secrets/kubernetes.io/serviceaccount/namespace - - - - -

2024-10-08 05:16:12.175Z Looking for service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace]. - - - - -

2024-10-08 05:16:12.175Z Service account namespace value: /var/run/secrets/kubernetes.io/serviceaccount/namespace - - - - -

2024-10-08 05:16:12.175Z Looking for service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace]. - - - - -

2024-10-08 05:16:12.175Z Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace]. - - - - -

2024-10-08 05:16:12.175Z Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace]. - - - - -

2024-10-08 05:16:12.199Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:12.199Z Config Map normalized sources : [{ config-map name : 'Optional[wxc-dhruva-proxy-static-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }, { config-map name : 'Optional[wxc-dhruva-proxy-dynamic-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }] - - - - -

2024-10-08 05:16:22.204Z Operation: [list] for kind: [ConfigMap] with name: [null] in namespace: [wxc-dhruva-mno] failed.. Ignoring. - - - - -

2024-10-08 05:16:22.205Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:22.246Z The single property with name: [dynamic.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:16:22.246Z Found source with name : 'wxc-dhruva-proxy-dynamic-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:22.246Z Loaded all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:22.252Z paths property sources : [] - - - - -

2024-10-08 05:16:22.252Z Secret name has not been set, taking it from property/env spring.application.name (default=application) - - - - -

2024-10-08 05:16:22.253Z Secrets normalized sources : [{ secret name : 'Optional[dhruvaProxyApplication]', namespace : 'Optional.empty'] - - - - -

2024-10-08 05:16:23.440Z Event: refreshAll detected - - - - -

2024-10-08 05:16:35.788Z Polling for changes in config maps - - - - -

2024-10-08 05:16:35.788Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:16:35.788Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:16:35.788Z environment from findPropertySources: ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap..wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:16:35.788Z findPropertySources : [configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno, configmap..wxc-dhruva-mno, configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno] - - - - -

2024-10-08 05:16:35.788Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:35.788Z environment sources from findPropertySources : [ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap..wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}] - - - - -

2024-10-08 05:16:35.788Z Config Map normalized sources : [{ config-map name : 'Optional[wxc-dhruva-proxy-static-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }, { config-map name : 'Optional[wxc-dhruva-proxy-dynamic-config-map]', namespace : 'Optional[wxc-dhruva-mno]', prefix : '[ConfigUtils.Prefix@659b8aaa name = 'DEFAULT']' }] - - - - -

2024-10-08 05:16:35.788Z bootstrap delegate class : class org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource - - - - -

2024-10-08 05:16:35.805Z Loaded all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:35.805Z Found source with name : 'wxc-dhruva-proxy-static-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:35.805Z The single property with name: [static.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:16:35.807Z The single property with name: [dynamic.yaml] will be treated as a yaml file - - - - -

2024-10-08 05:16:35.807Z Loaded (from cache) all config maps in namespace 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:35.807Z configmap namespace : wxc-dhruva-mno - - - - -

2024-10-08 05:16:35.807Z Found source with name : 'wxc-dhruva-proxy-dynamic-config-map in namespace: 'wxc-dhruva-mno' - - - - -

2024-10-08 05:16:35.812Z paths property sources : [] - - - - -

2024-10-08 05:16:35.812Z environment from locateMapPropertySources : ApplicationServletEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[ {name='spring-app-name'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap..wxc-dhruva-mno'}, BootstrapPropertySource {name='bootstrapProperties-configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, JndiPropertySource {name='jndiProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [bootstrap.yaml]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='applicationConfig: [classpath:/bootstrap.yaml]'}, MapPropertySource {name='springCloudDefaultProperties'}, OriginTrackedMapPropertySource {name='common-identity-client.yaml'}, OriginTrackedMapPropertySource {name='rate-limit-properties.yaml'}, OriginTrackedMapPropertySource {name='auto-configuration-properties.yaml'}, OriginTrackedMapPropertySource {name='service-health-properties.yaml'}, ResourcePropertySource {name='class path resource [git.properties]'}, OriginTrackedMapPropertySource {name='dynamic-logging-properties.yaml'}, CompositePropertySource {name='core-server-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-server-properties.yaml'}]}, CompositePropertySource {name='core-metrics-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='core-metrics-properties.yaml'}]}, CompositePropertySource {name='filter-properties.yaml', propertySources=[OriginTrackedMapPropertySource {name='filter-properties.yaml'}]}, OriginTrackedMapPropertySource {name='diagnostics-service-properties.yaml'}, OriginTrackedMapPropertySource {name='common-service-properties.yaml'}, {name='Management Server'}]} - - - - -

2024-10-08 05:16:35.813Z sources from locateMapPropertySources : [Fabric8ConfigMapPropertySource@981209569 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.####(MASKED)... - - - - -

**2024-10-08 05:16:35.815Z left size: 2 - - - - -

2024-10-08 05:16:35.817Z Fabric8ConfigMapPropertySource@981209569 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.#####(MASKED)... - - - - -

2024-10-08 05:16:35.817Z Fabric8ConfigMapPropertySource@452241362 {name='configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno', properties={#####(MASKED)} - - - - -

2024-10-08 05:16:35.817Z Fabric8ConfigMapPropertySource@85413271 {name='configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno', properties={common.####(MASKED)... - - - - -

2024-10-08 05:16:35.817Z Fabric8ConfigMapPropertySource@445069998 {name='configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno', properties={#####(MASKED)} - - - - -

2024-10-08 05:16:35.817Z right size: 3 - - - - -

2024-10-08 05:16:35.817Z Fabric8ConfigMapPropertySource@913738438 {name='configmap..wxc-dhruva-mno', properties={}} - - - - -

2024-10-08 05:16:35.818Z The current number of ConfigMap PropertySources does not match the ones loaded from Kubernetes - No reload will take place**

[wind57]

hoooly moly! @ryanjbaxter this is a bug, please re-open this one if there is such an option and add the bug label. Thank you.

These logs have been invaluable, without them diagnosing this issue would have been impossible.

I'll try to summarize what happens:

• user has configured two configmaps for property sources in namespace a, let's call them one and two
• When the app starts they are both read just fine and properties are populated correctly.
• At the same time, we create two property sources in Springs environment: configmap.one.a and configmap.two.a
• user enabled polling reload, so we will be polling k8s on a periodic bases to see if there are changes
• at some point in time, one of the calls to k8s fails (like user reported above), but we do not fail, instead, we log and move on:

In NamedSourceData (failsFast is false)

catch (Exception e) {
      onException(failFast, e);
}

We then try to find the names of the sources we read previously. Since we never failed above and only logged the issue, there will be no such names. We kind of do something like this:

    public static void main(String[] args) {

        new NamedSourceData() {
            @Override
            public MultipleSourcesContainer dataSupplier(LinkedHashSet<String> sourceNames) {
                return null;
            }
        }.go();

    }

    void go() {

        MultipleSourcesContainer data = MultipleSourcesContainer.empty();
        String names = data.names().stream().sorted()
            .collect(Collectors.joining(PROPERTY_SOURCE_NAME_SEPARATOR));
        SourceData sourceData = new SourceData(generateSourceName("configmap", names, "namespace", new String[]{}), data.data());

        System.out.println(sourceData);

    }

And the above prints: SourceData[sourceName=configmap..namespace, sourceData={}]

And this is exactly what the OP sees:

configmap..wxc-dhruva

This was the first mystery that was solved today.

But the above does not cause the issue, the real problem is that we will register this property source in Spring environment, though, so at the end of this operation there are 3 property sources:

configmap.one.a
configmap.two.a
configmap..a

while in k8s there are only first two... SO every next cycle of polling will fail, since the size of them are different, all the time. There is no way to recover from this.

---

I will think how to properly fix this in the upcoming days and present a PR.

[gmrammohan15]

@wind57 Really glad to know this.Had seen the logException code which just logs and moves on, but wasn't clear why empty source data remains persistent in all the future checks.Now based on your explanation its clear.Thanks much. Looking forward for your fix.

[wind57]

unfortunately, I'm at square zero here, after looking and the code and logs far closer, my explanation above only makes partial sense and the logs tend do disagree with my analysis. I will take a fresh look over them in a few days, may be something pops up.

[gmrammohan15]

Sure @wind57 . Just curious , which part of code and log is not aligning with analysis.

[wind57]

finally I had some time to put up a test to reproduce this issue, sorry it takes so much and I am so slow at this, this unfortunately is not my main job and I have little time these days...

Anyway, I did reproduce it via a test. It is indeed a bug on our side, I am positive about this. The core issue is still in the fact that when there is an error, we generate an incorrect name. The thing you need to get an error twice and some very unlucky timings.

I am not going to bother you with details, I need to think on how to solve this one now. :)

[gmrammohan15]

np, i understand. Really thanks for taking time for looking into this.Its a complex one , based on the symptoms i did try introducing few error scenarios to reproduce locally but couldn't.

[wind57]

I have a PR in motion to fix this, but I found a few improvements that we should make while investigating. As soon as those are reviewed and approved, I'll create the fix.

For the history, here is what happens.

In your current environment you have two property sources:

configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno

configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno

At some point, polling reload kicks in and we go to the k8s to see if there are any changes. While trying to read configmaps, we fail (that is the error you see with OK http3 that I explained). At that point we generate and incorrect name for a property source:

configmap..wxc-dhruva-mno

But, this one fails while reading the first configmap, it passes for the second one. So we end up with two property sources in environment and two coming from k8s, then we try to compare them and of course they differ (we compare contents not names). So... we say - "refresh".

Well, now when you call refresh, you reload property sources and fail again (you can see that "Operation: [list] for kind: [ConfigMap] with name: [null] in namespace: [wxc-dhruva-mno] failed.. Ignoring" twice), thus we generate in incorrect source name again. The difference now, is that unlike previously, this source is put into your Spring environment. So you end up with 3 sources that Spring knows about:

configmap.wxc-dhruva-proxy-dynamic-config-map.wxc-dhruva-mno

configmap.wxc-dhruva-proxy-static-config-map.wxc-dhruva-mno

configmap..wxc-dhruva-mno

Every next cycle compares these 3 to the 2 from the cluster and will always fail.

[wind57]

I have a PR in motion here, working on it...