RejectedExecutionException floods logs over failed listSyncAndWatch

michalvavrik commented 4 months ago

Describe the bug

I keep getting RejectedExecutionException over listSyncAndWatch and its unclear to me what should I do. It doesn't seem to affect my application in any way. It fills all my logs as the error is logged (sometimes) every ~ seconds and in all the tests I have.

Fabric8 Kubernetes Client version

6.13.0

Steps to reproduce

git clone git@github.com:quarkus-qe/quarkus-test-suite.git
cd quarkus-test-suite/http/http-minimum-reactive
mvn clean verify -Dopenshift -Dit.test=OpenShiftHttpMinimumReactiveIT

Expected behavior

Exception should be user friendly (tell me what is going on) or silent.

Runtime

OpenShift

Kubernetes API Server version

other (please specify in additional context)

Environment

other (please specify in additional context)

Fabric8 Kubernetes Client Logs

2024-06-25 13:59:47,434 ERROR [io.fab.kub.cli.inf.imp.cac.Reflector] (vert.x-eventloop-thread-9) listSyncAndWatch failed for v1/namespaces/ts-lmjzgculcl/pods, will stop: java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
    at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
    at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
    at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$null$1(VertxHttpRequest.java:121)
    at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
    at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
    at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
    at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
    at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:421)
    at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
    at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157)
    at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:709)
    at io.vertx.core.impl.ContextImpl.lambda$execute$4(ContextImpl.java:315)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.RejectedExecutionException
    at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47)
    at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179)
    at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136)
    at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116)
    at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    ... 30 more

Additional context

OpenShift version 4.15.15 which has Kubernetes version v1.28.9+416ecaf.

shawkins commented 4 months ago

Duplicate of #6059

michalvavrik commented 4 months ago

Duplicate of #6059

Thanks, looking at https://github.com/fabric8io/kubernetes-client/issues/6059 it really makes sense to try with reproducer, title and description of the issue.

manusa commented 4 months ago

A patch will be provided for 6.13.0 since this is a very nasty issue log-wise.

michalvavrik commented 4 months ago

A patch will be provided for 6.13.0 since this is a very nasty issue log-wise.

I'm watching, thanks. I missed there was #6059 when opening this issue.

fabric8io / kubernetes-client