CPU Spike is observed in the SpringBoot App when a Kubernetes configmap property is changed. The Springboot app is using the io.fabric8.kubernetes-client for reading the k8s configmap updates. Any help/ suggestions to fix the CPU spike

[dasari-rc]

Describe the bug The SpringBoot App deployed as POD in GCP Kubernetes. The Configurations are stored in Configmap & Secrets. The app is using io.fabric8.kubernetes-client as client to read the changes in Kubernetes Configmap. We are using @ConfgurationProperties Beans to read the properties from Configmap. We are seeing a huge spike in configmap when the property in configmap is changed real-time.

Sample

My Code snippets below:

We have the SpringBoot app boostrap.properties file as follows: spring.cloud.kubernetes.config.sources[0].namespace=default spring.cloud.kubernetes.config.sources[0].name=configmap-name1 spring.cloud.kubernetes.config.sources[1].namespace=default spring.cloud.kubernetes.config.sources[1].name=configmap-name2 spring.cloud.kubernetes.config.sources[2].namespace=default spring.cloud.kubernetes.config.sources[2].name=configmap-name3 endpoints.restart.enabled=true spring.cloud.kubernetes.reload.enabled=true spring.cloud.kubernetes.reload.strategy=refresh spring.cloud.kubernetes.reload.mode=event

io.fabric8 kubernetes-client 5.11.2 org.yaml snakeyaml com.fasterxml.jackson.dataformat jackson-dataformat-yaml

import lombok.Data; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.Configuration;

@Data @Configuration @ConfigurationProperties public class K8sConfigmap{ public String property1; public String proerpty2;

@Data
public static class  Prefix{
    public String subproperty1;
   public String subproperty2;
}

[ryanjbaxter]

What version of Spring Cloud?

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[dasari-rc]

What version of Spring Cloud?

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

We are using SpringBoot 2.7.1 and Spring Cloud Version 2021.0.3. We are using the following dependencies for Kubernetes Client:

org.springframework.cloud spring-cloud-starter-kubernetes-fabric8-config io.fabric8 kubernetes-client

    <dependency>
        <groupId>io.fabric8</groupId>
        <artifactId>kubernetes-client</artifactId>
        <version>5.11.2</version>
    </dependency>

We are using following bootstrap.properties: spring.cloud.kubernetes.reload.enabled=true spring.cloud.kubernetes.reload.strategy=refresh spring.cloud.kubernetes.enabled=true spring.cloud.kubernetes.reload.mode=event

[wind57]

you're not going to like the answer, but this version is not supported anymore...

Can you upgrade to 2023.x.x and see it it makes a difference?

[dasari-rc]

Also @ryanjbaxter i was observing one issue not sure if i can attribute it towards the CPU spike.

We have SpringBoot Microservices PODs running in GKE cluster for different domains. We have created 2 to 3 Configmap for each Microservice for externalizing the configurations. We have configured each Microservice to access these 2 to 3 dedicated Configmaps in the bootstrap.properties file using the following properties -

• spring.application.name=microservice1
• spring.cloud.kubernetes.enabled=true
• spring.cloud.kubernetes.reload.enabled=true
• spring.cloud.kubernetes.reload.mode=event
• spring.cloud.kubernetes.reload.strategy=refresh
• spring.cloud.kubernetes.config.sources[0].namespace=dev
• spring.cloud.kubernetes.config.sources[0].name=configmap-application-common
• spring.cloud.kubernetes.config.sources[1].namespace=dev
• spring.cloud.kubernetes.config.sources[1].name=configmap-address-validation-dev
• spring.cloud.kubernetes.config.sources[2].namespace=dev
• spring.cloud.kubernetes.config.sources[2].name=configmap-application-dev

We have created a service account which can have access ( get, list watch) to all the configmaps. For each Microservice we have assigned the service account created in the above step.

We are observing whenever a Realtime update is made to a configmap which is not part of the Microservice bootstrap.properties, still the class EventBasedConfigMapChangeDetector from org.springframework.cloud.kubernetes.fabric8.config.reload package is getting an event in the Microservice and is watching for the event and trying check for any property change which shouldnt have happened. As we have many PODs spinned up in PROD we are aseeing the CPU usage is crossing Limits and getting restarted.

Can you please help us here.

[ryanjbaxter]

As @wind57 said 2021.0.x is no longer supported please update to 2023.0.x and see if it make a difference

[wind57]

I'm going to re-iterate the same thing again that the version you are using is un-supported, so even if there is an issue there, there will not be a fix available.

To that extent, there are several reasons why you want to upgrade also:

• we switched to Informers from Watchers, which is recommended by the fabric8 team. There were various issues with watchers and all of them fixed with the usage of informers.
• we have added a property: spring.cloud.kubernetes.reload.enableReloadFiltering (false by default). This property allows selective informers to be created, it allows you to say : "I want to watch changes only for configmaps that have the label : spring.cloud.kubernetes.config.informer.enabled=true".

So instead of getting events from all configmaps in the namespace (what you are seeing in the version that you are using), you can get (by enabling spring.cloud.kubernetes.reload.enableReloadFiltering=true) changes only from some configmaps from the namespace (the ones that have a label: spring.cloud.kubernetes.config.informer.enabled=true). Seems like in your case, that would narrow the number of events to be processed in the pods and could solve your issue.

The last point is that might be related to the CPU, but might as well be not. After you upgrade and try out, may be profile the application to see what the real cause of the CPU spike is.

[wind57]

@ryanjbaxter can you add the feedback label back please? thank you

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.