search

fabric8io / kubernetes-client

Java client for Kubernetes & OpenShift
http://fabric8.io
Apache License 2.0
3.39k stars 1.46k forks source link

Leader not electing - Getting the exception : java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/V1Alpha1AdmissionRegistrationAPIGroupDSL #5733

Closed athulyashyju closed 3 months ago

athulyashyju commented 7 months ago

Describe the bug

Tried running the leader election example [https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/LeaderElectionExamples.java] with java version 17, Springboot version 3.1.4 kubernetes-client versions 6.10.0 and 6.2.0.

Outcome of 6.10.0: No leaders are elected

Outcome of 6.2.0: Leaders are electing and lease details are updating as expected but in-between getting exception like 'Unable to update LeaseLock '

Fabric8 Kubernetes Client version

6.10.0

Steps to reproduce

Same code as in https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/LeaderElectionExamples.java only lease lock is updating

try (KubernetesClient kc = new KubernetesClientBuilder().build()) {
                final Function<String, Lock> lockSupplier;
                lockSupplier = id -> new LeaseLock(NAMESPACE, NAME, id);
                new TestLeaderElectionSampleCode(kc, lockSupplier).run();
}

Expected behavior

Resolve the exception.

Runtime

other (please specify in additional context)

Kubernetes API Server version

other (please specify in additional context)

Environment

Windows, Amazon

Fabric8 Kubernetes Client Logs

With kubernetes-client version 6.10.0 getting the below error and no leaders are electing.

`{"@timestamp":"2024-01-30T10:20:13.435921952Z","@version":"1","message":"Application run failed","logger_name":"org.springframework.boot.SpringApplication","thread_name":"main","level":"ERROR","level_value":40000,"stack_trace":"org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testchatProcessor': Invocation of init method failed\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:222)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)\n\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)\n\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)\n\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)\n\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)\n\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)\n\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)\n\tat io.test.connector.jConnectorApplication.main(jConnectorApplication.java:22)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:95)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\nCaused by: java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/V1Alpha1AdmissionRegistrationAPIGroupDSL\n\tat java.base/java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.base/java.lang.ClassLoader.defineClass(Unknown Source)\n\tat java.base/java.security.SecureClassLoader.defineClass(Unknown Source)\n\tat java.base/java.net.URLClassLoader.defineClass(Unknown Source)\n\tat java.base/java.net.URLClassLoader$1.run(Unknown Source)\n\tat java.base/java.net.URLClassLoader$1.run(Unknown Source)\n\tat java.base/java.security.AccessController.doPrivileged(Unknown Source)\n\tat java.base/java.net.URLClassLoader.findClass(Unknown Source)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:149)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat java.base/java.lang.Class.getDeclaredConstructors0(Native Method)\n\tat java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)\n\tat java.base/java.lang.Class.getConstructor0(Unknown Source)\n\tat java.base/java.lang.Class.getConstructor(Unknown Source)\n\tat io.fabric8.kubernetes.client.KubernetesClientBuilder.build(KubernetesClientBuilder.java:79)\n\tat io.test..EventProcessor.configureTopicSubscription(EventProcessor.java:121)\n\tat io.test..EventProcessor.configureTopicSubscription(EventProcessor.java:97)\n\tat io.test..impl.testchatProcessor.init(testchatProcessor.java:76)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219)\n\t... 26 common frames omitted\nCaused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.client.V1Alpha1AdmissionRegistrationAPIGroupDSL\n\tat java.base/java.net.URLClassLoader.findClass(Unknown Source)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:149)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\t... 52 common frames omitted\n","appname":"jconnector"}`

But in the version 6.2.0, the leader was electing and completing the code but in-between getting an exception like below. I was not able to catch the exception.

`{"@timestamp":"2024-01-29T12:23:35.215188288Z","@version":"1","message":"Exception occurred while acquiring lock 'LeaseLock: default - concurrent-example-lease (Concurrent-24)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"-1792101289-pool-13-thread-7","level":"ERROR","level_value":40000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to update LeaseLock\n\tat io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock.update(LeaseLock.java:102)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:209)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$acquire$4(LeaderElector.java:151)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$8(LeaderElector.java:253)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.20.0.1/apis/coordination.k8s.io/v1/namespaces/default/leases/concurrent-example-lease. Message: Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-example-lease\": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=concurrent-example-lease, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-example-lease\": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).\n\tat io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:517)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:551)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleUpdate(OperationSupport.java:347)\n\tat io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleUpdate(BaseOperation.java:680)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$replace$0(HasMetadataOperation.java:167)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:172)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:113)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:41)\n\tat io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:1043)\n\tat io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:88)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock.update(LeaseLock.java:100)\n\t... 7 common frames omitted\nCaused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.20.0.1/apis/coordination.k8s.io/v1/namespaces/default/leases/concurrent-example-lease. Message: Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-example-lease\": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=concurrent-example-lease, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-example-lease\": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:709)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:689)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:640)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:576)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:618)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)\n\tat io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$4.onResponse(OkHttpClientImpl.java:277)\n\tat okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)\n\t... 3 common frames omitted\n","appname":"jconnector"}`

How can i solve this error? Thanks in advance!

Additional context

Client Version: v1.28.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.5-eks-5e0fdde

Java version : 17 Springboot version: 3.1.4

manusa commented 7 months ago

This seems more related to Spring Boot than to the client.

If you're explicitly creating the KubernetesClient instance, I'm not sure why Spring is expecting a KubernetesClient bean to be defined or where and why it's trying to inject it.

Are you sure you don't have any @autowired or @Inject annotations in your TestLeaderElectionSampleCode class?

Also, have you tried defining the bean so that you can reuse the client in your application and properly close it once the application is finished?

athulyashyju commented 7 months ago

@manusa Thanks for the response. In the kubernetes-client version 6.10.0, i have tried defining the bean and using it in application, but now it is failing in the kubernetesClientBuilder. but the same code was electing leader for the version 6.2.0. please find the below logs for reference:

{"@timestamp":"2024-01-30T10:20:13.435921952Z","@version":"1","message":"Application run failed","logger_name":"org.springframework.boot.SpringApplication","thread_name":"main","level":"ERROR","level_value":40000,"stack_trace":"org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testchatProcessor': Invocation of init method failed\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:222)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)\n\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)\n\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)\n\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)\n\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)\n\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)\n\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)\n\tat io.test.connector.jConnectorApplication.main(jConnectorApplication.java:22)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:95)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\nCaused by: java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/V1Alpha1AdmissionRegistrationAPIGroupDSL\n\tat java.base/java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.base/java.lang.ClassLoader.defineClass(Unknown Source)\n\tat java.base/java.security.SecureClassLoader.defineClass(Unknown Source)\n\tat java.base/java.net.URLClassLoader.defineClass(Unknown Source)\n\tat java.base/java.net.URLClassLoader$1.run(Unknown Source)\n\tat java.base/java.net.URLClassLoader$1.run(Unknown Source)\n\tat java.base/java.security.AccessController.doPrivileged(Unknown Source)\n\tat java.base/java.net.URLClassLoader.findClass(Unknown Source)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:149)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat java.base/java.lang.Class.getDeclaredConstructors0(Native Method)\n\tat java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)\n\tat java.base/java.lang.Class.getConstructor0(Unknown Source)\n\tat java.base/java.lang.Class.getConstructor(Unknown Source)\n\tat io.fabric8.kubernetes.client.KubernetesClientBuilder.build(KubernetesClientBuilder.java:79)\n\tat io.test..EventProcessor.configureTopicSubscription(EventProcessor.java:121)\n\tat io.test..EventProcessor.configureTopicSubscription(EventProcessor.java:97)\n\tat io.test..impl.testchatProcessor.init(testchatProcessor.java:76)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401)\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219)\n\t... 26 common frames omitted\nCaused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.client.V1Alpha1AdmissionRegistrationAPIGroupDSL\n\tat java.base/java.net.URLClassLoader.findClass(Unknown Source)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:149)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\t... 52 common frames omitted\n","appname":"jconnector"}
athulyashyju commented 7 months ago

Added a more focused error description and title

manusa commented 7 months ago

Are you by any chance using Spring Cloud Kubernetes?

Check the FAQ: https://github.com/fabric8io/kubernetes-client/blob/main/doc/FAQ.md#ive-tried-adding-a-dependency-to-kubernetes-client-but-im-still-getting-weird-class-loading-issues-what-gives

athulyashyju commented 7 months ago

Thanks for sharing the link but we are using spring cloud openfeign dependencies Spring Cloud Kubernetes not, does it affect?

manusa commented 7 months ago

I'm assuming that Spring Cloud OpenFeign has a transitive dependency to Spring Cloud core and the Fabric8 module, so extremely likely this is the problem.

athulyashyju commented 7 months ago

Hi @manusa , I have tested the code with only the below dependencies, but still getting the same error java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/V1Alpha1AdmissionRegistrationAPIGroupDSL

<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.tomcat.embed</groupId>
                    <artifactId>tomcat-embed-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.tomcat.embed</groupId>
                    <artifactId>tomcat-embed-el</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.tomcat.embed</groupId>
                    <artifactId>tomcat-embed-websocket</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-client</artifactId>
            <version>6.10.0</version>
        </dependency>
        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
        </dependency>
        <dependency>
            <groupId>jakarta.xml.bind</groupId>
            <artifactId>jakarta.xml.bind-api</artifactId>
            <version>2.3.2</version>
        </dependency>

please find below the maven dependency tree


 +- org.springframework.boot:spring-boot-starter-web:jar:3.1.4:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:3.1.4:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:3.1.4:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:3.1.4:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:3.1.4:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.4.11:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.4.11:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.20.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.20.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:2.0.9:compile
[INFO] |  |  +- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile
[INFO] |  |  +- org.springframework:spring-core:jar:6.0.12:compile
[INFO] |  |  |  \- org.springframework:spring-jcl:jar:6.0.12:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.33:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:3.1.4:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.2:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.2:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.15.2:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:3.1.4:compile
[INFO] |  +- org.springframework:spring-web:jar:6.0.12:compile
[INFO] |  |  +- org.springframework:spring-beans:jar:6.0.12:compile
[INFO] |  |  \- io.micrometer:micrometer-observation:jar:1.11.4:compile
[INFO] |  |     \- io.micrometer:micrometer-commons:jar:1.11.4:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:6.0.12:compile
[INFO] |     +- org.springframework:spring-aop:jar:6.0.12:compile
[INFO] |     +- org.springframework:spring-context:jar:6.0.12:compile
[INFO] |     \- org.springframework:spring-expression:jar:6.0.12:compile
[INFO] +- io.fabric8:kubernetes-client:jar:6.10.0:compile
[INFO] |  +- io.fabric8:kubernetes-client-api:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-core:jar:6.2.0:compile
[INFO] |  |  |  \- io.fabric8:kubernetes-model-common:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-gatewayapi:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-rbac:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-admissionregistration:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-apps:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-autoscaling:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-apiextensions:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-batch:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-certificates:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-coordination:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-discovery:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-events:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-extensions:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-flowcontrol:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-networking:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-metrics:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-policy:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-scheduling:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-storageclass:jar:6.2.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-node:jar:6.2.0:compile
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.15.2:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.15.2:compile
[INFO] |  +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.2.0:runtime
[INFO] |  |  +- com.squareup.okhttp3:okhttp:jar:4.10.0:runtime
[INFO] |  |  |  +- com.squareup.okio:okio-jvm:jar:3.0.0:runtime
[INFO] |  |  |  |  \- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.8.22:runtime
[INFO] |  |  |  \- org.jetbrains.kotlin:kotlin-stdlib:jar:1.8.22:runtime
[INFO] |  |  |     \- org.jetbrains:annotations:jar:13.0:runtime
[INFO] |  |  \- com.squareup.okhttp3:logging-interceptor:jar:4.10.0:runtime
[INFO] |  |     \- org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.8.22:runtime
[INFO] |  |        \- org.jetbrains.kotlin:kotlin-stdlib-jdk7:jar:1.8.22:runtime
[INFO] |  \- io.fabric8:zjsonpatch:jar:0.3.0:compile
[INFO] +- net.logstash.logback:logstash-logback-encoder:jar:7.4:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.15.2:compile
[INFO] |     \- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2:compile
[INFO] +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] |  \- jakarta.activation:jakarta.activation-api:jar:2.1.2:compile
[INFO] +- org.springdoc:springdoc-openapi-starter-webmvc-ui:jar:2.2.0:compile
[INFO] |  +- org.springdoc:springdoc-openapi-starter-webmvc-api:jar:2.2.0:compile
[INFO] |  |  \- org.springdoc:springdoc-openapi-starter-common:jar:2.2.0:compile
[INFO] |  |     \- io.swagger.core.v3:swagger-core-jakarta:jar:2.2.15:compile
[INFO] |  |        +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] |  |        +- io.swagger.core.v3:swagger-annotations-jakarta:jar:2.2.15:compile
[INFO] |  |        +- io.swagger.core.v3:swagger-models-jakarta:jar:2.2.15:compile
[INFO] |  |        \- jakarta.validation:jakarta.validation-api:jar:3.0.2:compile
[INFO] |  \- org.webjars:swagger-ui:jar:5.2.0:compile
[INFO] +- org.slf4j:slf4j-api:jar:2.0.9:compile
[INFO] \- junit:junit:jar:4.13.2:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:2.2:test
[INFO]       \- org.hamcrest:hamcrest:jar:2.2:test
manusa commented 7 months ago

Check how the model and API modules are referencing an older 6.2.0 version while the client is at 6.10.0.

Please add the BOM dependency as the FAQ states or override all of the dependency definitions.

Also, note that maybe Spring Cloud might not work with 6.10.0 at all, so you might want to wait until they release a new version with a newer Kubernetes Client dependency version.

athulyashyju commented 7 months ago

When i added the BOM dependency the model and API modules are updated to 6.10.0.

After adding the BOM, single thread leader election works with kubernetes-client 6.10.0 but when i test concurrent thread example, i am still getting an exception like below:

Can you tell me what the cause of this exception is?

{"@timestamp":"2024-02-01T17:15:26.643036914Z","@version":"1","message":"Exception occurred while acquiring lock 'LeaseLock: default - concurrent-leader-lease (Concurrent-79) retrying...'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"-1333938290-pool-2-thread-80","level":"ERROR","level_value":40000,"stack_trace":"io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/default/leases/concurrent-leader-lease. Message: Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-leader-lease\": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=concurrent-leader-lease, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-leader-lease\": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).\n\tat io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)\n\tat io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:763)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:252)\n\tat io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:45)\n\tat io.fabric8.kubernetes.client.extension.ResourceAdapter.patch(ResourceAdapter.java:306)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ResourceLock.update(ResourceLock.java:65)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:236)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$acquire$4(LeaderElector.java:172)\n\tat io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$8(LeaderElector.java:282)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/default/leases/concurrent-leader-lease. Message: Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-leader-lease\": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=concurrent-leader-lease, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io \"concurrent-leader-lease\": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589)\n\tat io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)\n\tat io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:143)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)\n\tat io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)\n\tat io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:137)\n\t... 3 common frames omitted\n","appname":"test-test-connector"}
athulyashyju commented 7 months ago

Hi @manusa ,

I am facing some issue for the single thread leader election, Scenerio 1:

        CompletableFuture<?> startLeaderElection = leader.start();
        Thread.sleep(10000); 
       // startLeaderElection.cancel(true); //commented this i want the leader be active

when i have scaled the pod to one, the leader got elected. But when i scaled to 2 (without scaling down) the leader is electing in newly created pod even when the old pod is still a leader. The lease have old pod as holderidentity at first but then new leader is electing with new pod details and the lease is updated with new pod details.

PS: The leader election in first pod is not stopped/killed.

Scenerio 2:

            CompletableFuture<?> startLeaderElection = leader.start();
            Thread.sleep(10000); 
           startLeaderElection.cancel(true); //uncommented

I have first scaled the pod to 1 and then to 2. The leader election happened and stopped in both the pods.

My expectation is: When i scaled to one, the first pod should be the leader and should be active until some interruptions happens or the pod is killed. And then i scaled to two , the first pod still be the leader, no leader should elect in the second pod because the leader is active in first pod. The leader election in second pod should fail.

Do you have any suggestions on how to achieve my expectations or if I am doing it wrong?

shawkins commented 7 months ago

Can you tell me what the cause of this exception is?

Conflicts are expected during leader election. They are a result of multiple electors competing. Logging this at an error level is misleading and was addressed by https://github.com/fabric8io/kubernetes-client/pull/5747

when i have scaled the pod to one, the leader got elected. But when i scaled to 2 (without scaling down) the leader is electing in newly created pod even when the old pod is still a leader.

More than likely you have given the same identity to both electors. You need to include something unique to each elector in the identity.

athulyashyju commented 7 months ago

Hi @shawkins , Thanks for the response. The identity i have given is pod name so both the pods have different identity. For reference please find below

first pod : test-connector-58987494fb-j2mv9 Logs: New leader elected test-connector-58987494fb-j2mv9

scaled to 2: first pod: test-connector-58987494fb-j2mv9
second pod: test-connector-58987494fb-scwzm Logs: New leader elected test-connector-58987494fb-j2mv9 New leader elected test-connector-58987494fb-scwzm

When scaled to 2, the logs are showing both the pod names.

shawkins commented 7 months ago

@athulyashyju having different identities helps rule out that they are simply taking over for each other. However there is not enough information here to determining what is happening - please provide the full details of what's in your LeaderElectionConfig and the pod logs.

If a lease has been obtained, a new leader will only be possible if the lease has not been renewed within the duration.

athulyashyju commented 7 months ago

@shawkins Please find below the config i am using and lease details.

For the microservice, we needed to process multiple events. So i am passing each event name as lease name. So multiple lease will be created for one pod. When leader is elected for each event the code for processing the events will run.

final String NAMESPACE = "default";
final String NAME = eventName;
try (KubernetesClient kc = new KubernetesClientBuilder().build()) {
    LeaderElector leader  = kc.leaderElector().withConfig(new LeaderElectionConfigBuilder().withReleaseOnCancel()
            .withName("Leader Election configuration").withLeaseDuration(Duration.ofSeconds(15L))
            .withLock(new LeaseLock(NAMESPACE, NAME, lockIdentity)).withRenewDeadline(Duration.ofSeconds(10L))
            .withRetryPeriod(Duration.ofSeconds(2L))
            .withLeaderCallbacks(new LeaderCallbacks(() -> System.out.println("STARTED LEADERSHIP"), () -> {
                System.out.println("STOPPED LEADERSHIP");
            }, newLeader -> {
                ApiClient client = null;
                try {
                    client = Config.defaultClient();
                } catch (IOException e) {
                    e.printStackTrace();
                }
                Configuration.setDefaultApiClient(client);
                CoordinationV1Api coordinationApi1 = new CoordinationV1Api();
                V1Lease lease;
                try {
                    lease = coordinationApi1.readNamespacedLease(NAME, NAMESPACE, null);
                    System.out.println("Current Lease\n" + lease);
                } catch (ApiException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }  
               /////......
                // Internal code for processing the events
    })).build()).build();
    CompletableFuture<?> startLeaderElection = leader.start();
}

Lease details

class V1Lease {
    apiVersion: coordination.k8s.io/v1
    kind: Lease
    metadata: class V1ObjectMeta {
        annotations: null
        creationTimestamp: 2024-02-08T19:21:34Z
        deletionGracePeriodSeconds: null
        deletionTimestamp: null
        finalizers: null
        generateName: null
        generation: null
        labels: null
        managedFields: [class V1ManagedFieldsEntry {
            apiVersion: coordination.k8s.io/v1
            fieldsType: FieldsV1
            fieldsV1: {f:spec={f:acquireTime={}, f:holderIdentity={}, f:leaseDurationSeconds={}, f:leaseTransitions={}, f:renewTime={}}}
            manager: fabric8-kubernetes-client
            operation: Update
            subresource: null
            time: 2024-02-08T19:30:48Z
        }]
        name: chat-event
        namespace: default
        ownerReferences: null
        resourceVersion: 1347099
        selfLink: null
        uid: 0b2df84b-2f8-409c-8027-801ff7bc80cd
    }
    spec: class V1LeaseSpec {
        acquireTime: 2024-02-08T19:30:47.007418Z
        holderIdentity: test-connector-58987494fb-j67mq
        leaseDurationSeconds: 15
        leaseTransitions: 1
        renewTime: 2024-02-08T19:30:47.007418Z
    }
}
shawkins commented 7 months ago

You are creating the client / elector in a try catch block that will autoclose the client and stop the elector when it exits, is there some kind of wait or other logic not shown that would keep the client alive after your call to leader.start()

If you do think that you are keeping things alive appropriately, can you provide a more detailed / debug set of log from each pod showing that each pod is obtaining the same lease?

Also, why are you creating what appears to be a new client to read the lease - you can just reuse the kc client instance for that?

stale[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!