Open vinkenfl opened 9 months ago
1nb0und
commented
9 months ago @vinkenfl thanks for reporting the issue. I will be releasing 8.3.4 soon with the added bump for grpc-netty as a workaround so you are not blocked.
For now, feel free to use this latest release candidate https://github.com/camunda-community-hub/spring-zeebe/releases/tag/8.3.2-rc4
Let us know if you still face issues
vinkenfl
commented
9 months ago Even rc4 shows the same issues:
My dependencies:
io.camunda.spring:spring-boot-starter-camunda:8.3.2-rc4 io.camunda.spring:spring-client-zeebe:8.3.2-rc4 io.camunda:zeebe-client-java:8.3.3 io.grpc:grpc-api:1.58.0 io.grpc:grpc-core:1.58.0 io.grpc:grpc-netty:1.58.0 io.grpc:grpc-stub:1.58.0 io.camunda:zeebe-gateway-protocol-impl:8.3.3 io.grpc:grpc-api:1.58.0 ....
It seems that RC4 is still using 1.58.0 instead of 1.59.1 as intended by merge #543
Call stack
io.grpc.StatusRuntimeException: UNKNOWN
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNKNOWN
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52)
at com.<omitted>.zeebe.util.Utilities.deployResources(Utilities.java:74)
at com.<omitted>.zeebe.util.Utilities.deployResource(Utilities.java:58)
at com.<omitted>.zeebe.ZeebeProductOrderProcessTest.deployProcessTest(ZeebeProductOrderProcessTest.java:132)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
at app//io.grpc.Status.asRuntimeException(Status.java:537)
at app//io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
at app//io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
at app//io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
at app//io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
at app//io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
at app//io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at app//io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.9/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.UnsupportedOperationException
at io.grpc.netty.AbstractHttp2Headers.isEmpty(AbstractHttp2Headers.java:40)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:419)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:352)
at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188)
at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
adrianAzoitei
commented
9 months ago Can confirm the issue is still present on 8.3.2-rc4.
1nb0und
commented
9 months ago Reopening this ticket and will investigate this
vinkenfl
commented
9 months ago Enforcing version 1.59.1 results in other exceptions for my testcases. My dependencies
io.camunda.spring:spring-boot-starter-camunda-test:8.3.2-rc4
io.camunda:zeebe-process-test-extension:8.3.3
io.camunda:zeebe-process-test-engine:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-protobuf:1.58.0
io.grpc:grpc-stub:1.58.0
io.camunda.spring:spring-boot-starter-camunda:8.3.2-rc4
io.camunda.spring:spring-client-zeebe:8.3.2-rc4
io.camunda:zeebe-client-java:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-core:1.59.1
io.grpc:grpc-netty:1.59.1
io.grpc:grpc-stub:1.58.0
io.camunda:zeebe-gateway-protocol-impl:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-protobuf:1.58.0
io.grpc:grpc-stub:1.58.0 The new callstack
java.lang.reflect.UndeclaredThrowableException
at jdk.proxy3/jdk.proxy3.$Proxy268.waitForIdleState(Unknown Source)
at com.<omitted>.zeebe.util.Utilities.waitForIdleState(Utilities.java:128)
at com.<omitted>.zeebe.util.Utilities.startProcessInstance(Utilities.java:97)
at com.<omitted>.zeebe.ZeebeProductOrderProcessTest.runDeletePathTest(ZeebeProductOrderProcessTest.java:477)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.camunda.zeebe.spring.test.proxy.ZeebeTestEngineProxy.handleInvocation(ZeebeTestEngineProxy.java:33)
at io.camunda.zeebe.spring.test.proxy.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:40)
... 7 more
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at io.camunda.zeebe.process.test.engine.InMemoryEngine.waitForIdleState(InMemoryEngine.java:134)
... 13 more@vinkenfl @adrianAzoitei Please try 8.3.4-rc1 when you get a chance. This version ensures everything uses the fixed version of grpc.
I noticed on 8.3.2-rcs, there are conflicting versions due to some dependencies are on 8.3.3, and others are at 8.3.2. This causes a mismatch and incompatibility between grpc libraries.
Let us know if you still face any issues
vinkenfl
commented
9 months ago Sorry to say that, but I still have issues even with 8.3.4-rc1. In the dependencies you still see some 1.58.0 parts:
io.camunda.spring:spring-boot-starter-camunda-test:8.3.4-rc1
io.camunda:zeebe-process-test-extension:8.3.3
io.camunda:zeebe-process-test-engine:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-protobuf:1.58.0
io.grpc:grpc-stub:1.58.0
io.camunda.spring:spring-boot-starter-camunda:8.3.4-rc1
io.camunda.spring:spring-client-zeebe:8.3.2-rc4
io.camunda:zeebe-client-java:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-core:1.59.1
io.grpc:grpc-netty:1.59.1
io.grpc:grpc-stub:1.58.0
io.camunda:zeebe-gateway-protocol-impl:8.3.3
io.grpc:grpc-api:1.59.1
io.grpc:grpc-protobuf:1.58.0
io.grpc:grpc-api:1.59.1
io.grpc:grpc-protobuf-lite:1.58.0
io.grpc:grpc-api:1.59.1
io.grpc:grpc-stub:1.58.0My callstack now looks like this:
java.lang.reflect.UndeclaredThrowableException
at jdk.proxy3/jdk.proxy3.$Proxy339.waitForIdleState(Unknown Source)
at com.<omitted>.zeebe.util.Utilities.waitForIdleState(Utilities.java:128)
at com.<omitted>.zeebe.util.Utilities.startProcessInstance(Utilities.java:97)
at com.<omitted>.zeebe.ZeebeCancelProductOrderProcessTest.runSunnyDayPathTest(ZeebeCancelProductOrderProcessTest.java:85)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.camunda.zeebe.spring.test.proxy.ZeebeTestEngineProxy.handleInvocation(ZeebeTestEngineProxy.java:33)
at io.camunda.zeebe.spring.test.proxy.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:40)
... 7 more
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at io.camunda.zeebe.process.test.engine.InMemoryEngine.waitForIdleState(InMemoryEngine.java:134)
... 13 more@vinkenfl Thanks for reporting the issue. I have a mistake on my part and was incorrectly overriding the dependency. As part of this recent PR, i then made a release for 8.3.4-rc5
Please try 8.3.4-rc5 and I also verified on my end thru a separate test project and can confirm it now uses 1.60.0 for io.grpc
Let us know how it goes.
@adrianAzoitei feel free to use the new version so that you avoid overriding it yourself in your project
vinkenfl
commented
9 months ago Nope. Dependencies:
io.camunda.spring:spring-boot-starter-camunda-test-common:8.3.4-rc5
io.grpc:grpc-api:1.60.0
io.grpc:grpc-core:1.60.0
io.grpc:grpc-netty:1.60.0
io.grpc:grpc-protobuf-lite:1.60.0
io.grpc:grpc-protobuf:1.60.0
io.grpc:grpc-stub:1.60.0
io.grpc:grpc-util:1.60.0Call stack:
java.lang.reflect.UndeclaredThrowableException
at jdk.proxy3/jdk.proxy3.$Proxy339.waitForIdleState(Unknown Source)
at com.<omitted>.util.Utilities.waitForIdleState(Utilities.java:128)
at com.<omitted>.zeebe.util.Utilities.startProcessInstance(Utilities.java:97)
at com.<omitted>.zeebe.ZeebeCancelProductOrderProcessTest.runSunnyDayPathTest(ZeebeCancelProductOrderProcessTest.java:85)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.camunda.zeebe.spring.test.proxy.ZeebeTestEngineProxy.handleInvocation(ZeebeTestEngineProxy.java:33)
at io.camunda.zeebe.spring.test.proxy.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:40)
... 7 more
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at io.camunda.zeebe.process.test.engine.InMemoryEngine.waitForIdleState(InMemoryEngine.java:134)
... 13 more
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.camunda.zeebe.spring.test.proxy.ZeebeTestEngineProxy.handleInvocation(ZeebeTestEngineProxy.java:33)
at io.camunda.zeebe.spring.test.proxy.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:40)
at jdk.proxy3/jdk.proxy3.$Proxy339.waitForIdleState(Unknown Source)
at com.telekom.conan.pom.zeebe.util.Utilities.waitForIdleState(Utilities.java:128)
at com.telekom.conan.pom.zeebe.util.Utilities.startProcessInstance(Utilities.java:97)
at com.telekom.conan.pom.zeebe.ZeebeCancelProductOrderProcessTest.runSunnyDayPathTest(ZeebeCancelProductOrderProcessTest.java:85)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at io.camunda.zeebe.process.test.engine.InMemoryEngine.waitForIdleState(InMemoryEngine.java:134)
... 13 more@vinkenfl
1nb0und
commented
9 months ago Hello @vinkenfl, any sample code (doesn't have to be a full project when you get a chance) you can share where the error can be reproduced will be really appreciated!
Running SpringBoot 3.2 together with io.camunda.spring:spring-boot-starter-camunda:8.3.2-rc1 I have the following exceptions during job execution:
This is likely to be caused by io/grpc/netty/AbstractHttp2Headers.java where the isEmpty() method is not implemented for version 1.58 of io.grpc:grpc-netty.
This is a bug in grpc-netty which is fixed in version 1.59.1 (see https://github.com/grpc/grpc-java/releases). Please consider an update to this version.