search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.76k stars 2.68k forks source link

RejectedExecutionException in fabric8 when checking status of amq-streams operator #28405

Closed fedinskiy closed 2 years ago

fedinskiy commented 2 years ago

Describe the bug

I have an application, which deploys Kafka on Openshift via amq-streams operator, checks it status, connects to it via quarkus-smallrye-reactive-messaging-kafka and when deletes it. It uses fabric8 library to check status of the operator. This works flawlessly with Quarkus 2.13.0.Final, but fails when running with 999-SNAPSHOT

Expected behavior

The scenario should work without failures.

Actual behavior

The scenario fails with this stacktrace:

SEVERE Caught exception while closing extension context: org.junit.jupiter.engine.descriptor.ClassExtensionContext@50b46e24: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Kafka]  with name: [kafka-instance]  in namespace: [ts-yffbyetnae]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:177)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:139)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:88)
    at io.quarkus.test.bootstrap.inject.OpenShiftClient.isCustomResourceReady(OpenShiftClient.java:503)
    at io.quarkus.test.services.operator.OperatorManagedResource.lambda$customResourcesAreReady$0(OperatorManagedResource.java:87)
    at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
    at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
    at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
    at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
    at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:533)
    at io.quarkus.test.services.operator.OperatorManagedResource.customResourcesAreReady(OperatorManagedResource.java:87)
    at io.quarkus.test.services.operator.OperatorManagedResource.isRunning(OperatorManagedResource.java:53)
    at io.quarkus.test.bootstrap.BaseService.isRunning(BaseService.java:129)
    at io.quarkus.test.bootstrap.BaseService.stop(BaseService.java:210)
    at io.quarkus.test.bootstrap.BaseService.close(BaseService.java:227)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
    at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:68)
    at org.junit.jupiter.engine.descriptor.AbstractExtensionContext.close(AbstractExtensionContext.java:80)
    at org.junit.jupiter.engine.execution.JupiterEngineExecutionContext.close(JupiterEngineExecutionContext.java:53)
    at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:222)
    at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:57)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$cleanUp$10(NodeTestTask.java:167)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.cleanUp(NodeTestTask.java:167)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:98)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.io.InterruptedIOException: executor rejected
    at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157)
    at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204)
    at okhttp3.Dispatcher.enqueue(Dispatcher.java:144)
    at okhttp3.RealCall.enqueue(RealCall.java:93)
    at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172)
    ... 70 more
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@51c65a43 rejected from java.util.concurrent.ThreadPoolExecutor@740dcae3[Shutting down, pool size = 2, active threads = 0, queued tasks = 0, completed tasks = 165]
    at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
    at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
    at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
    at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:154)
    ... 80 more

How to Reproduce?

  1. git clone git@github.com:quarkus-qe/quarkus-test-suite.git
  2. Connect to openshift installation and verify, that oc binary exists on $PATH
  3. Run test: mvn clean verify -Popenshift -pl messaging/kafka-streams-reactive-messaging/ -Dall-modules -Dit.test=OperatorOpenShiftAmqStreamsKafkaStreamIT. It should fail.
  4. Run the same test, but with 2.13: mvn clean verify -Popenshift -pl messaging/kafka-streams-reactive-messaging/ -Dall-modules -Dit.test=OperatorOpenShiftAmqStreamsKafkaStreamIT -Dquarkus.platform.version=2.13.0.Final. It should succeed.

Output of uname -a or ver

5.19.11-200.fc36.x86_64

Output of java -version

11.0.16

GraalVM version (if different from Java)

No response

Quarkus version or git rev

6dad45448433bbe1b902d3ac3db6ea4782195197

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)

Additional information

$ oc version
Client Version: 4.11.0-202208020706.p0.g7075089.assembly.stream-7075089
Kustomize Version: v4.5.4
Kubernetes Version: v1.24.0+3882f8f
quarkus-bot[bot] commented 2 years ago

/cc @Sgitario, @geoand, @iocanel

Sgitario commented 2 years ago

Can you share the Quarkus application logs as well? I suspect the issue is in OpenShiftClient.isCustomResourceReady, not in Quarkus. Perhaps, it should be reported in the Fabric8 Kubernetes Client @manusa

fedinskiy commented 2 years ago

@Sgitario application logs for failed and successful run are here: https://gist.github.com/fedinskiy/7e9f9c6ec7ef56296c44837b2831ee28

The only meaningful difference I was able to find is difference between versions of dependencies and one new dependency: org.eclipse.microprofile.metrics.microprofile-metrics-api-3.0.1.jar

UPD: different dependencies were added to the gist

Sgitario commented 2 years ago

Since the application starts ok, I dont think this is caused by those new dependencies or Quarkus, but an issue either in your Quarkus QE Testing framework (something you need to change to use the new Fabric8 Kubernetes Client) or in Fabric8 Kubernetes

geoand commented 2 years ago

I agree with @Sgitario. My guess is that something is different in Kubernetes Client 6.x (cc @manusa)

manusa commented 2 years ago

It certainly doesn't look like an error caused by a change of the microprofile-metrics-api dependency version.

I need to clone github.com:quarkus-qe/quarkus-test-suite.git and further investigate what might be the issue. Right now I can only say that it looks like the CRD is not properly configured. We did change the behavior for non-standard/CRD APIs but I can't recall if we implemented the check to verify that a CRD existed before allowing operations on that API resource.

manusa commented 2 years ago

BTW, is this running on a real cluster or on the Mock Server? For some reason I assumed the latter, but now I realize that might not be the case.

manusa commented 2 years ago

This dependency has an unaligned version of the client.

...
+- io.quarkus.qe:quarkus-test-openshift:jar:1.2.0.Beta12:test
[INFO] |  +- io.fabric8:openshift-client:jar:5.12.3:test
[INFO] |  |  +- io.fabric8:kubernetes-client:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-core:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-rbac:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-admissionregistration:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-apps:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-autoscaling:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-apiextensions:jar:5.12.3:test
[INFO] |  |  |  +- io.fabric8:kubernetes-model-batch:jar:5.12.3:test
...

I'm not sure if this will fix the issue, but the quarkus-test-openshift project needs to be updated.

Sgitario commented 2 years ago

Many thanks @manusa ! I'm closing this issue.