fabric8.kubernetes.client.KubernetesClientException - Detection of Kubernetes version failed.

[peppe77]

K8S: v1.18

Strimzi Kafka Operator was v0.18 - operational Kafka Cluster v2.5.0 - operational

In order for us to get to Kafka cluster 2.7.0, we first need to upgrade operator to v0.22 though came across problem.

Upgraded operator from 0.18 to 0.22 and got following:

io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:54) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.fabric8.kubernetes.client.DefaultKubernetesClient.getVersion(DefaultKubernetesClient.java:489) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
        at io.strimzi.operator.PlatformFeaturesAvailability.lambda$getVersionInfoFromKubernetes$5(PlatformFeaturesAvailability.java:150) ~[io.strimzi.operator-common-0.22.0.jar:0.22.0]
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: okhttp3.internal.http2.ConnectionShutdownException

This could have been caused by different number of reasons (not necessarily client release) though most obvious ones we ruled out for following reasons:

1. We had successfully upgraded operator from 0.15 to 0.18 and did not experience a problem with K8S client is not able to reach K8S api;
2. We did some manual tests / checks to ensure there was nothing preventing client from reaching K8S api
3. [most critical one]: we set HTTP2_DISASBLE: "true" and strimzi operator passed through the point it used to fail. POD actually comes up and before it would cyclic crash/loop. (there are other errors in the logs but related to kafka clusters and objects we have - nothing related to the issue herein reported).

Could this be a problem with lib/java version used by the K8S client? Asking that because there were 2-3 previously reported issues (java8, if not mistaken) with similar symptom and also "addressed" by this very same work-around.

If you need additional logs/information, please let us know. All we need to do to reproduce the problem is remove env variable passed (HTTP2_DISABLE) as mentioned above. Just wanted some help as to further troubleshoot this problem.

[peppe77]

Just update here - enabled additional log in K8S Api Server and as you can see below, it TCP RST the client attempts with the following:

kube-apiserver-ip-10-64-1-194.ec2.internal kube-apiserver I0603 04:10:09.138948 1 log.go:172] http2: server rejecting conn: INADEQUATE_SECURITY, Prohibited TLS 1.2 Cipher Suite: 9d kube-apiserver-ip-10-64-1-194.ec2.internal kube-apiserver I0603 04:10:09.138978 1 log.go:172] http2: Framer 0xc014c8b880: wrote GOAWAY len=43 LastStreamID=0 ErrCode=INADEQUATE_SECURITY Debug="Prohibited TLS 1.2 Cipher Suite: 9d"

This happens with v5.0.2 though based on below (very similar issues) - this should have been fixed in v5.0.2 already, no?

https://github.com/fabric8io/kubernetes-client/issues/2212 https://github.com/kubernetes-client/java/issues/1149

Please advise. thanks /Pedro

[manusa]

this should have been fixed in v5.0.2 already, no?

That issue should have already been fixed for any 5.x version.

But you could further rule this out by using a >8 Java version.

[peppe77]

@manusa the Strimzi kafka operator v0.22 is on Java11. Could this be caused due to the cipher proposed that is not allowed in TLS v1.2 (there is a long list that is not supported) ? Any suggestions on how to better understand and isolate the problem? thanks in advance

[peppe77]

@manusa i know what is going on. Check this out...

1. Based on -> https://www.ibm.com/docs/en/zos/2.3.0?topic=programming-cipher-suite-definitions , 9d => TLS_RSA_WITH_AES_256_GCM_SHA384 which based on -> https://httpwg.org/specs/rfc7540.html#BadCipherSuites is considered weak and should not be proposed.
2. We have "forced" our K8S 1.18/API-Server to accept this specific cipher and problem disappeared as expected. In our clusters, for security reasons, we do not allow weak ciphers (TLS v1.2) that have been blacklisted.

Conclusions:

• client (fabric8io) shall not propose, by default, such cipher - the default option should be any cipher that is not weak/blacklisted like the currently used 9d - see above link.
• the Strimzi/operator needs to either handle that somehow (we opened bug report with them as well) OR place a requirement to have client properly handle that. Ideally client should pick a cipher that by default has not been blacklisted.
• Anyone that has such combination strimzi kafka operator v0.22/K8S 1.18 and works is because api-server is not properly configured [that is -> allows a weak/black listed cipher to be used]

for reference: have opened this on Strimzi/Kafka: https://github.com/strimzi/strimzi-kafka-operator/issues/5044

[manusa]

As a workaround, maybe the changes contributed in #3171(#3170) can be used to create an OkHttpClient with an overridden list of protocols, and then pass this client to the DefaultKubernetsClient constructor.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[Vlatombe]

According to https://github.com/square/okhttp/blob/998633be00d1b2952d068ea04b376fd83bc05c3f/okhttp/src/main/kotlin/okhttp3/ConnectionSpec.kt#L302-L310, these ciphers are forbidden when using TLS1.2 and http/2, so the server rejects them. Kubernetes-client defaults to MODERN_TLS, I think that to accomodate this use case it should use RESTRICTED_TLS, at least when using http/2.

[rsoni43]

hi guys,

but I am getting the exact error atm, what was the fix around this..

I am on Strimzi 0.18.0 and bumping to 0.22.0 K8s - 1.21.1

2023-01-04 13:53:22 INFO  Main:60 - ClusterOperator 0.22.0 is starting
2023-01-04 13:53:23 ERROR PlatformFeaturesAvailability:152 - Detection of Kubernetes version failed.
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:54) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.getVersion(DefaultKubernetesClient.java:489) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.strimzi.operator.PlatformFeaturesAvailability.lambda$getVersionInfoFromKubernetes$5(PlatformFeaturesAvailability.java:150) ~[io.strimzi.operator-common-0.22.0.jar:0.22.0]
    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: okhttp3.internal.http2.ConnectionShutdownException
    at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:247) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:230) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http2.Http2Codec.writeRequestHeaders(Http2Codec.java:117) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:50) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:42) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:148) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.handleVersionGet(ClusterOperationsImpl.java:62) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:46) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    ... 8 more
2023-01-04 13:53:23 ERROR Main:89 - Failed to gather environment facts
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:54) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.getVersion(DefaultKubernetesClient.java:489) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.strimzi.operator.PlatformFeaturesAvailability.lambda$getVersionInfoFromKubernetes$5(PlatformFeaturesAvailability.java:150) ~[io.strimzi.operator-common-0.22.0.jar:0.22.0]
    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: okhttp3.internal.http2.ConnectionShutdownException
    at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:247) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:230) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http2.Http2Codec.writeRequestHeaders(Http2Codec.java:117) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:50) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:42) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:148) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:93) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.handleVersionGet(ClusterOperationsImpl.java:62) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:46) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
    ... 8 more
Stream closed EOF for infrastructure/strimzi-cluster-operator-6947d474d7-rgg2z (strimzi-cluster-operator)

[lishalongfa]

hello, I got similar error. my k8s version is v1.24.3. may I ask how to fix it?

2023-02-16 09:37:56.839  INFO --- [           main] c.d.daivb.core.DaivbSpringApplication    : Started Spring Boot application in the regular mode.
2023-02-16 09:39:05.420  INFO --- [nio-8011-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-02-16 09:39:05.424  INFO --- [nio-8011-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 3 ms
2023-02-16 10:32:33.086  WARN --- [://10.0.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : ConfigMaps watch closed

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 206515220 (451772264)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:258)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273)
    at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

2023-02-16 11:22:15.770  WARN --- [://10.0.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : ConfigMaps watch closed

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 245868605 (451801551)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:258)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273)
    at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[shawkins]

@lishalongfa that is not the same error. That will happen on watches occasionally if the server lacks bookmark support. You are supposed to handle this situation in the Watcher.onClose(WatcherException) method or use an Informer, which handles this automatically.

[lishalongfa]

hello @shawkins, is this watches integrated into k8s of cloud? like if my service was deployed on Azure k8s, does this watches was integrated into Azure k8s? which means I do not have the right to change it?

[shawkins]

@lishalongfa please refer to https://kubernetes.io/docs/reference/using-api/api-concepts/ to learn more about how watches may terminate. Also please convert any further questions to a separate discussion topic so they are not associated with this older issue.

[lishalongfa]

@shawkins okay and thx. I've never seen it before and then know little about it. It looks like very similar for me as it's also a KubernetesClientException and related to version and fabric8.

[manusa]

Is this issue still applicable? Does switching to a different HTTP Client implementation (such as Vert.x) fix the issue?