search

fabric8io / kubernetes-client

Java client for Kubernetes & OpenShift
http://fabric8.io
Apache License 2.0
3.41k stars 1.46k forks source link

informer not invoking the event callbacks anymore #4416

Closed howshit closed 2 years ago

howshit commented 2 years ago

Describe the bug

Hi there,

we are using io.fabric8:kubernetes-client:5.10.2 to connect to kubernetes clusters. we've defined some CRDs, and use the informer from fabric8 to attach callbacks to the add, update and delete events of the CRs. and we've encountered some issues recently, the callbacks are not invoked anymore after the api server gets restarted(sometime the reconnect is successfully, and sometimes the reconnect fails), and the error message is like:

2022-09-14 16:17:43.834 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.io.EOFException null 2022-09-14 16:17:43.834 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.io.EOFException null 2022-09-14 16:17:43.834 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.io.EOFException null 2022-09-14 16:17:49.840 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:17:49.840 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:17:49.840 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:17:56.849 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:17:56.849 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:17:56.849 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:18:05.855 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:18:05.855 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:18:05.855 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure java.net.SocketTimeoutException Read timed out 2022-09-14 16:18:15.586 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake 2022-09-14 16:18:15.586 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake 2022-09-14 16:18:15.586 WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake Exception in thread "OkHttp Dispatcher" Exception in thread "OkHttp Dispatcher" Exception in thread "OkHttp Dispatcher" java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688) at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592) at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688) at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592) at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688) at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592) at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)

then our logic inside the callbacks(for add, update and delete event) are not invoked anymore.

Would you please kindly help on this?

Best regards, Felix

Fabric8 Kubernetes Client version

5.10.2

Steps to reproduce

  1. run an application using this version of fabric8 kubernetes client, use informer to listen to the changes of CRs.
  2. restart the api server(until the informer is not working anymore)

Expected behavior

the client is able to reconnect after the api server gets back.

Runtime

other (please specify in additional context)

Kubernetes API Server version

1.22

Environment

macOS

Fabric8 Kubernetes Client Logs


2022-09-14 16:17:43.834  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.io.EOFException null
2022-09-14 16:17:43.834  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.io.EOFException null
2022-09-14 16:17:43.834  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.io.EOFException null
2022-09-14 16:17:49.840  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:17:49.840  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:17:49.840  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:17:56.849  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:17:56.849  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:17:56.849  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:18:05.855  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:18:05.855  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:18:05.855  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure java.net.SocketTimeoutException Read timed out
2022-09-14 16:18:15.586  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake
2022-09-14 16:18:15.586  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake
2022-09-14 16:18:15.586  WARN 76867 --- [ternal:6443/...] i.f.k.c.d.i.WatcherWebSocketListener     : Exec Failure javax.net.ssl.SSLHandshakeException Remote host terminated the handshake
Exception in thread "OkHttp Dispatcher" Exception in thread "OkHttp Dispatcher" Exception in thread "OkHttp Dispatcher" java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688)
    at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69)
    at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688)
    at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69)
    at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Status.getMessage()" because "status" is null
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:688)
    at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:69)
    at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

Additional context

No response

shawkins commented 2 years ago

This bug is specific to 5.10.2. It was introduced by some refinements to the general requestFailure handling. I believe any version starting with 5.11 addresses this.