search

smallrye / smallrye-reactive-messaging

SmallRye Reactive Messaging
http://www.smallrye.io/smallrye-reactive-messaging/
Apache License 2.0
235 stars 176 forks source link

Vert.x buffer overflow when sending AMQP messages #1541

Closed belks closed 2 years ago

belks commented 2 years ago

Hi,

we encountered some strange issues with Vert.x when using smallrye-reactive-messaging and ActiveMQ - see exception below. I hope you can help or point us in the right direction if we are doing something completely wrong.

Uncaught exception received by Vert.x - Exception: newPosition > limit: (65536 > 32768) - java.lang.IllegalArgumentException: newPosition > limit: (65536 > 32768) at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) at java.base/java.nio.Buffer.position(Buffer.java:316) at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:267) at org.apache.qpid.proton.engine.impl.FrameWriterBuffer.transferTo(FrameWriterBuffer.java:215) at org.apache.qpid.proton.engine.impl.FrameWriter.readBytes(FrameWriter.java:73) at org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:376) at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59) at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.pending(SaslImpl.java:842) at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.pending(HandshakeSniffingTransportWrapper.java:138) at org.apache.qpid.proton.engine.impl.TransportImpl.pending(TransportImpl.java:1591) at org.apache.qpid.proton.engine.impl.TransportImpl.getOutputBuffer(TransportImpl.java:1540) at io.vertx.proton.impl.ProtonTransport.flush(ProtonTransport.java:273) at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:305) at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:294) at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:952) at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:919) at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50) at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274) at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22) at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53) at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22) at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:942) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)

About our system. We are in the process of splitting up a monolithic component into smaller independent pieces. For that purpose we added:

All data is sent as plain byte arrays in the messages.

The in-memory ActiveMQ, the emitter and all consumers are running in the same JVM (zulu17.28.13-ca-jdk17.0.0-win_x64 locally and Zulu17.28+14-SA build 17+35-LTS on Openshift) because we are still a transitioning phase as mentioned above.

Our application is running on Openshift, but the issue also happens locally on our Windows machines. However, it does not happen deterministically. Sometimes we encounter the issue immediately when processing data, sometimes only after a few hours.

After the Vert.x exception the application continues to run, but the messages presented to the consumers by smallrye-reactive-messaging then contain incomplete byte arrays, which then causes our application to throw exceptions.

Versions: artemis-server 2.17.0 artemis-amqp-protocol 2.17.0 quarkus-universe-bom 2.4.0.Final quarkus-smallrye-reactive-messaging-amqp 2.4.0.Final

Our config:

mp:
  messaging:
    outgoing:
      queue-A-out:
        connector: smallrye-amqp
        address: A-queue
        durable: true
        tracing-enabled: false
      queue-B-out:
        connector: smallrye-amqp
        address: B-queue
        durable: true
        tracing-enabled: false
      queue-C-out: 
        connector: smallrye-amqp
        address: C-queue
        durable: true
        tracing-enabled: false
    incoming:
      queue-A-in: 
        connector: smallrye-amqp
        address: A-queue
        durable: false
        tracing-enabled: false
      queue-B-in:
        connector: smallrye-amqp
        address: B-queue
        durable: false
        tracing-enabled: false
      queue-C-in:
        connector: smallrye-amqp
        address: C-queue
        durable: false
        tracing-enabled: false

ActiveMQ acceptor URL:

tcp://127.0.0.1:5672?jms.copyMessageOnSend=false&jms.prefetchPolicy.all=1000&amqpMinLargeMessageSize=1048576&jms.useAsyncSend=true&jms.optimizeAcknowledge=true&jms.redeliveryPolicy.maximumRedeliveries=-1&compressLargeMessages=false&protocols=AMQP&jms.useCompression=false

Our code:

Emitter sending to queue A:


    @Inject
    public Creator(@Channel("queue-A-out") @OnOverflow(value = NONE) Emitter<byte[]> emitter) {
        this.emitter = emitter;
    }

   public void produce(OurMessage message) {
        // used by multiple theads at the same time. But should be fine. Emitter.send() is synchronized
        emitter.send(message.toBytes()).toCompletableFuture().join();
    }

Consumer consuming queue A and sending data to queues B and C:

    private final Emitter<byte[]> emitterQueueB;
    private final Emitter<byte[]> emitterQueueC;

    @Inject
    public Transformer(
            @Channel("queue-B-out") @OnOverflow(value = NONE) Emitter<byte[]> emitterQueueB,
            @Channel("queue-C-out") @OnOverflow(value = NONE) Emitter<byte[]> emitterQueueC) {
        // ... usual constructor stuff
    }

    @Incoming("queue-A-in")
    @Blocking(value = "WorkerPool-A", ordered = false)
    public CompletionStage<Void> transform(Message<byte[]> message) {
           // process message
           byte[] messageData = .... // build new message
           CompletionStage<Void> firstMsg = emitterQueueB.send(messageData);
           CompletionStage<Void> secondMsg = emitterQueueC.send(messageData);
           firstMsg.thenCombine(secondMsg, (void1, void2) -> null).toCompletableFuture().join();
           return message.ack();
    }

Consumer consuming queue B:

    @Incoming("queue-B-in")
    @Blocking(value = "WorkerPool-B", ordered = false)
    public CompletionStage<Void> consume(Message<byte[]> message) {
         // process message
         return message.ack();
    }

Consumer consuming queue C:

    @Incoming("queue-C-in")
    @Blocking(value = "WorkerPool-C", ordered = false)
    public CompletionStage<Void> consume(Message<byte[]> message) {
        // process message
         return message.ack();
    }
cescoffier commented 2 years ago

@gemmellr any idea? It seems to be in Proton itself.

gemmellr commented 2 years ago

@cescoffier I'd have to go with illegal multi-threaded use of the engine by the bits using vertx-proton (whether that be vertx-amqp-client, smallryre-reactive-messaging-amqp, or above).

The bit it is in at the time of the exception is writing from an intermediate framing buffer into the transport output buffer, and it fails with an attempt to set the target buffer to its new position following an array copy of the payload...apparently trying to set it to beyond its limit, even though it checked how much was remaining before starting and so couldnt possibly do that by itself.

Specifically it fails trying to set the position to twice its limit, i.e another thread is in there at the same time doing the same thing and just filled the buffer and bumped the position by the size up to the limit itself. Then when this thread (the expected event loop thread for the connection, doing an idle timeout check) did so it unwittingly ends up trying to set it to beyond the limit it was already at.

cescoffier commented 2 years ago

Thanks @gemmellr,

I can see multiple @Blocking. While all the actions should happen on the correct event loop, we may miss something.

cescoffier commented 2 years ago

Hum, I've tried to reproduce it without any luck. I followed the given code snippet and configured the worker pools, but for me everything worked. Would it be possible to get a reproducer?

cescoffier commented 2 years ago

I even load-test it to be sure that the emitter are using the correct threads and it looks like it's alright (every access are made from the same event loop)

gemmellr commented 2 years ago

The particular stack above is from the periodic idle timeout check started on the connections event loop when the connection opens. Artemis advertises a default 30sec timeout I think, so I expect the check is kicking in every 15 seconds (it pessimistically uses half the peers advertised value as the timeout) as the connection tries to ensure its satisfying it, and its possibly only at that point it would likely go pop.

You could force the idle timeout processing to occur more frequently by either reducing the brokers advertised timeout, or configuring the client side with its own idle timeout (they are independent in each direction) thats smaller, either of which would make it process the check more often. I think the option is called 'heartbeat' to try and to try and avoid discussion of advertised-vs-actual timeout.

gemmellr commented 2 years ago

When you say the accesses are using the same event loop, is it the correct thread the connection requires, i.e the one doing the idle timeout check.

cescoffier commented 2 years ago

Yes, the connection happens on an event loop we capture, and everything is done on that event loop. However, I'm not sure about the idle timeout. That's a very good point. I will try to see if I reduce it to something very small. Unfortunately, I would not be able to use my dev service for this :-(

gemmellr commented 2 years ago

The idle timeout processing is scheduled on the event loop during a callback while processing the connection open.

cescoffier commented 2 years ago

Still unable to reproduce it. @belks, can you provide a reproducer?

belks commented 2 years ago

I will try

belks commented 2 years ago

Hi,

I uploaded a reproducer to https://github.com/belks/smallrye-reactive-messaging-issue-1541-reproducer/

Unfortunately it is still not happening deterministically. I had to run it several times to get the error again. I attached the full log of the successful test run with the reproducer at the end of this comment.

How to use it: 1) Run the application by running the StartMe.java 2) Go to http://localhost:8080/swagger-ui/ 3) Start a few producers by calling the /start endpoint with the amount of workers you want. I usually use 8.

C:\develop\tools\zulu11.2.3-jdk11.0.1-win_x64\bin\java.exe "-javaagent:C:\develop\tools\JetBrains\IntelliJ IDEA 2021.2.3\lib\idea_rt.jar=60942:C:\develop\tools\JetBrains\IntelliJ IDEA 2021.2.3\bin" -Dfile.encoding=UTF-8 -classpath C:\develop\workspace\MinimalReproducer\target\classes;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy\2.4.0.Final\quarkus-resteasy-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx-http\2.4.0.Final\quarkus-vertx-http-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-security-runtime-spi\2.4.0.Final\quarkus-security-runtime-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-mutiny\2.4.0.Final\quarkus-mutiny-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx-http-dev-console-runtime-spi\2.4.0.Final\quarkus-vertx-http-dev-console-runtime-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\security\quarkus-security\1.1.4.Final\quarkus-security-1.1.4.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-web\2.14.1\smallrye-mutiny-vertx-web-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-web-common\2.14.1\smallrye-mutiny-vertx-web-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-auth-common\2.14.1\smallrye-mutiny-vertx-auth-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-bridge-common\2.14.1\smallrye-mutiny-vertx-bridge-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-web\4.1.5\vertx-web-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-web-common\4.1.5\vertx-web-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-auth-common\4.1.5\vertx-auth-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-bridge-common\4.1.5\vertx-bridge-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-core\4.1.5\vertx-core-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-resolver-dns\4.1.68.Final\netty-resolver-dns-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-dns\4.1.68.Final\netty-codec-dns-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy-server-common\2.4.0.Final\quarkus-resteasy-server-common-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy-common\2.4.0.Final\quarkus-resteasy-common-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\resteasy\resteasy-core\4.7.0.Final\resteasy-core-4.7.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\spec\javax\ws\rs\jboss-jaxrs-api_2.1_spec\2.0.1.Final\jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\spec\javax\xml\bind\jboss-jaxb-api_2.3_spec\2.0.0.Final\jboss-jaxb-api_2.3_spec-2.0.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\resteasy\resteasy-core-spi\4.7.0.Final\resteasy-core-spi-4.7.0.Final.jar;C:\Users\tk8es\.m2\repository\com\ibm\async\asyncutil\0.1.0\asyncutil-0.1.0.jar;C:\Users\tk8es\.m2\repository\com\sun\activation\jakarta.activation\1.2.1\jakarta.activation-1.2.1.jar;C:\Users\tk8es\.m2\repository\jakarta\validation\jakarta.validation-api\2.0.2\jakarta.validation-api-2.0.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-openapi\2.4.0.Final\quarkus-smallrye-openapi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-open-api-core\2.1.15\smallrye-open-api-core-2.1.15.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\openapi\microprofile-openapi-api\2.0.1-RC1\microprofile-openapi-api-2.0.1-RC1.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.12.5\jackson-core-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.12.5\jackson-databind-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.12.5\jackson-annotations-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\dataformat\jackson-dataformat-yaml\2.12.5\jackson-dataformat-yaml-2.12.5.jar;C:\Users\tk8es\.m2\repository\org\jboss\jandex\2.4.1.Final\jandex-2.4.1.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-core\2.4.0.Final\quarkus-core-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\jakarta\enterprise\jakarta.enterprise.cdi-api\2.0.2\jakarta.enterprise.cdi-api-2.0.2.jar;C:\Users\tk8es\.m2\repository\jakarta\el\jakarta.el-api\3.0.3\jakarta.el-api-3.0.3.jar;C:\Users\tk8es\.m2\repository\jakarta\interceptor\jakarta.interceptor-api\1.2.5\jakarta.interceptor-api-1.2.5.jar;C:\Users\tk8es\.m2\repository\jakarta\inject\jakarta.inject-api\1.0\jakarta.inject-api-1.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-ide-launcher\2.4.0.Final\quarkus-ide-launcher-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-development-mode-spi\2.4.0.Final\quarkus-development-mode-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config\2.5.1\smallrye-config-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-core\2.5.1\smallrye-config-core-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-expression\1.6.0\smallrye-common-expression-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-function\1.6.0\smallrye-common-function-1.6.0.jar;C:\Users\tk8es\.m2\repository\org\jboss\logmanager\jboss-logmanager-embedded\1.0.9\jboss-logmanager-embedded-1.0.9.jar;C:\Users\tk8es\.m2\repository\org\jboss\logging\jboss-logging-annotations\2.2.1.Final\jboss-logging-annotations-2.2.1.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\threads\jboss-threads\3.4.2.Final\jboss-threads-3.4.2.Final.jar;C:\Users\tk8es\.m2\repository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;C:\Users\tk8es\.m2\repository\org\jboss\slf4j\slf4j-jboss-logmanager\1.1.0.Final\slf4j-jboss-logmanager-1.1.0.Final.jar;C:\Users\tk8es\.m2\repository\org\graalvm\sdk\graal-sdk\21.2.0\graal-sdk-21.2.0.jar;C:\Users\tk8es\.m2\repository\org\wildfly\common\wildfly-common\1.5.4.Final-format-001\wildfly-common-1.5.4.Final-format-001.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-bootstrap-runner\2.4.0.Final\quarkus-bootstrap-runner-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-io\1.6.0\smallrye-common-io-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-arc\2.4.0.Final\quarkus-arc-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\arc\arc\2.4.0.Final\arc-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\jakarta\transaction\jakarta.transaction-api\1.3.3\jakarta.transaction-api-1.3.3.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\context-propagation\microprofile-context-propagation-api\1.2\microprofile-context-propagation-api-1.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-swagger-ui\2.4.0.Final\quarkus-swagger-ui-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-config-yaml\2.4.0.Final\quarkus-config-yaml-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-source-yaml\2.5.1\smallrye-config-source-yaml-2.5.1.jar;C:\Users\tk8es\.m2\repository\org\yaml\snakeyaml\1.29\snakeyaml-1.29.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-common\2.5.1\smallrye-config-common-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-classloader\1.6.0\smallrye-common-classloader-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-constraint\1.6.0\smallrye-common-constraint-1.6.0.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\config\microprofile-config-api\2.0\microprofile-config-api-2.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging-amqp\2.4.0.Final\quarkus-smallrye-reactive-messaging-amqp-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-amqp\3.10.1\smallrye-reactive-messaging-amqp-3.10.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-amqp-client\2.14.1\smallrye-mutiny-vertx-amqp-client-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-amqp-client\4.1.5\vertx-amqp-client-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-proton\4.1.5\vertx-proton-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-api\1.6.0\opentelemetry-api-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-context\1.6.0\opentelemetry-context-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-semconv\1.6.0-alpha\opentelemetry-semconv-1.6.0-alpha.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-converter-api\2.6.0\smallrye-reactive-converter-api-2.6.0.jar;C:\Users\tk8es\.m2\repository\org\reactivestreams\reactive-streams\1.0.3\reactive-streams-1.0.3.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny\1.1.2\mutiny-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-mutiny-reactive-streams-operators\2.4.0.Final\quarkus-mutiny-reactive-streams-operators-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\reactive-streams-operators\microprofile-reactive-streams-operators-api\1.0.1\microprofile-reactive-streams-operators-api-1.0.1.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\reactive-streams-operators\microprofile-reactive-streams-operators-core\1.0.1\microprofile-reactive-streams-operators-core-1.0.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny-reactive-streams-operators\1.1.2\mutiny-reactive-streams-operators-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny-smallrye-context-propagation\1.1.2\mutiny-smallrye-context-propagation-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation\1.2.2\smallrye-context-propagation-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation-api\1.2.2\smallrye-context-propagation-api-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation-storage\1.2.2\smallrye-context-propagation-storage-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-context-propagation\2.4.0.Final\quarkus-smallrye-context-propagation-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging\2.4.0.Final\quarkus-smallrye-reactive-messaging-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging-kotlin\2.4.0.Final\quarkus-smallrye-reactive-messaging-kotlin-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-annotation\1.6.0\smallrye-common-annotation-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-health\3.10.1\smallrye-reactive-messaging-health-3.10.1.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\health\microprofile-health-api\3.1\microprofile-health-api-3.1.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx\2.4.0.Final\quarkus-vertx-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-netty\2.4.0.Final\quarkus-netty-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-http2\4.1.68.Final\netty-codec-http2-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-haproxy\4.1.68.Final\netty-codec-haproxy-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-core\2.14.1\smallrye-mutiny-vertx-core-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-runtime\2.14.1\smallrye-mutiny-vertx-runtime-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\vertx-mutiny-generator\2.14.1\vertx-mutiny-generator-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-codegen\4.1.5\vertx-codegen-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-fault-tolerance-vertx\5.2.1\smallrye-fault-tolerance-vertx-5.2.1.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-jackson\2.4.0.Final\quarkus-jackson-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.12.5\jackson-datatype-jsr310-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.12.5\jackson-datatype-jdk8-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.12.5\jackson-module-parameter-names-2.12.5.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-provider\3.10.1\smallrye-reactive-messaging-provider-3.10.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-api\3.10.1\smallrye-reactive-messaging-api-3.10.1.jar;C:\Users\tk8es\.m2\repository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-amqp-protocol\2.17.0\artemis-amqp-protocol-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-selector\2.17.0\artemis-selector-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-core-client\2.17.0\artemis-core-client-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\jgroups\jgroups\3.6.13.Final\jgroups-3.6.13.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-epoll\4.1.68.Final\netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-unix-common\4.1.68.Final\netty-transport-native-unix-common-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-kqueue\4.1.68.Final\netty-transport-native-kqueue-4.1.68.Final-osx-x86_64.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-http\4.1.68.Final\netty-codec-http-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-handler\4.1.68.Final\netty-handler-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-handler-proxy\4.1.68.Final\netty-handler-proxy-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec\4.1.68.Final\netty-codec-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-socks\4.1.68.Final\netty-codec-socks-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-common\4.1.68.Final\netty-common-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-commons\2.17.0\artemis-commons-2.17.0.jar;C:\Users\tk8es\.m2\repository\commons-beanutils\commons-beanutils\1.9.4\commons-beanutils-1.9.4.jar;C:\Users\tk8es\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Users\tk8es\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\tk8es\.m2\repository\org\apache\geronimo\specs\geronimo-json_1.0_spec\1.0-alpha-1\geronimo-json_1.0_spec-1.0-alpha-1.jar;C:\Users\tk8es\.m2\repository\org\jboss\logging\jboss-logging\3.4.2.Final\jboss-logging-3.4.2.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-server\2.17.0\artemis-server-2.17.0.jar;C:\Users\tk8es\.m2\repository\com\google\guava\guava\30.1.1-jre\guava-30.1.1-jre.jar;C:\Users\tk8es\.m2\repository\com\google\guava\failureaccess\1.0.1\failureaccess-1.0.1.jar;C:\Users\tk8es\.m2\repository\com\google\guava\listenablefuture\9999.0-empty-to-avoid-conflict-with-guava\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar;C:\Users\tk8es\.m2\repository\com\google\code\findbugs\jsr305\3.0.2\jsr305-3.0.2.jar;C:\Users\tk8es\.m2\repository\com\google\errorprone\error_prone_annotations\2.2.0\error_prone_annotations-2.2.0.jar;C:\Users\tk8es\.m2\repository\com\google\j2objc\j2objc-annotations\1.3\j2objc-annotations-1.3.jar;C:\Users\tk8es\.m2\repository\org\jboss\logmanager\jboss-logmanager\2.1.10.Final\jboss-logmanager-2.1.10.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-jdbc-store\2.17.0\artemis-jdbc-store-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-dbcp2\2.7.0\commons-dbcp2-2.7.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-pool2\2.7.0\commons-pool2-2.7.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\activemq-artemis-native\1.0.2\activemq-artemis-native-1.0.2.jar;C:\Users\tk8es\.m2\repository\org\jctools\jctools-core\2.1.2\jctools-core-2.1.2.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-configuration2\2.7\commons-configuration2-2.7.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-lang3\3.12.0\commons-lang3-3.12.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-text\1.8\commons-text-1.8.jar;C:\Users\tk8es\.m2\repository\io\micrometer\micrometer-core\1.7.4\micrometer-core-1.7.4.jar;C:\Users\tk8es\.m2\repository\org\hdrhistogram\HdrHistogram\2.1.12\HdrHistogram-2.1.12.jar;C:\Users\tk8es\.m2\repository\org\latencyutils\LatencyUtils\2.0.3\LatencyUtils-2.0.3.jar;C:\Users\tk8es\.m2\repository\commons-io\commons-io\2.11.0\commons-io-2.11.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-journal\2.17.0\artemis-journal-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\qpid\proton-j\0.33.10\proton-j-0.33.10.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-buffer\4.1.68.Final\netty-buffer-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport\4.1.68.Final\netty-transport-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-resolver\4.1.68.Final\netty-resolver-4.1.68.Final.jar com.minimal.reproducer.StartMe
Press [h] for more options>
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-12-13 12:05:27,649 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.log.console.json" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 12:05:27,649 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.smallrye-metrics.path" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 12:05:27,649 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.smallrye-health.root-path" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 12:05:29,485 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-A-in MDC={} 
2021-12-13 12:05:29,500 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-B-in MDC={} 
2021-12-13 12:05:29,500 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-C-in MDC={} 
2021-12-13 12:05:29,500 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-A-out MDC={} 
2021-12-13 12:05:29,516 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-B-out MDC={} 
2021-12-13 12:05:29,516 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-C-out MDC={} 
2021-12-13 12:05:29,547 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,547 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,547 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,547 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,563 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,563 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 12:05:29,563 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) ActiveMQ dir: C:\Users\tk8es\AppData\Local\Temp\1\activemq11469402518254187551 MDC={} 
2021-12-13 12:05:29,563 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) ActiveMQ connection properties: jms.copyMessageOnSend=false&jms.prefetchPolicy.all=1000&amqpMinLargeMessageSize=1048576&jms.useAsyncSend=true&jms.optimizeAcknowledge=true&jms.redeliveryPolicy.maximumRedeliveries=-1&compressLargeMessages=false&protocols=AMQP&jms.useCompression=false MDC={} 
2021-12-13 12:05:29,616 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) Starting embedded ActiveMQ MDC={} 
2021-12-13 12:05:29,886 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq11469402518254187551\journal,bindingsDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq11469402518254187551\bindings,largeMessagesDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq11469402518254187551\largeMessages,pagingDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq11469402518254187551\paging) MDC={} 
2021-12-13 12:05:29,886 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221045: libaio is not available, switching the configuration into NIO MDC={} 
2021-12-13 12:05:29,902 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221013: Using NIO Journal MDC={} 
2021-12-13 12:05:29,933 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 4’269’801’472 MDC={} 
2021-12-13 12:05:29,948 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE MDC={} 
2021-12-13 12:05:29,948 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP MDC={} 
2021-12-13 12:05:29,948 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP MDC={} 
2021-12-13 12:05:29,986 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601138: User anonymous@unknown is getting notification info on target resource: null [] MDC={} 
2021-12-13 12:05:29,986 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl@474ed82f [] MDC={} 
2021-12-13 12:05:29,986 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601138: User anonymous@unknown is getting notification info on target resource: ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [] MDC={} 
2021-12-13 12:05:30,002 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221034: Waiting indefinitely to obtain live lock MDC={} 
2021-12-13 12:05:30,002 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221035: Live Server Obtained live lock MDC={} 
2021-12-13 12:05:30,133 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@5133cf23 [] MDC={} 
2021-12-13 12:05:30,171 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AcceptorControlImpl@41e2f672 [] MDC={} 
2021-12-13 12:05:30,202 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221020: Started NIO Acceptor at 127.0.0.1:5672 for protocols [AMQP] MDC={} 
2021-12-13 12:05:30,202 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221007: Server is now live MDC={} 
2021-12-13 12:05:30,202 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.17.0 [localhost, nodeID=95746395-5c04-11ec-9cf8-0433c2e55376]  MDC={} 
2021-12-13 12:05:30,218 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@10ea91bd [] MDC={} 
2021-12-13 12:05:30,234 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-A in AddressSettings MDC={} 
2021-12-13 12:05:30,234 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-A in AddressSettings MDC={} 
2021-12-13 12:05:30,249 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@6070e85b [] MDC={} 
2021-12-13 12:05:30,265 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@265f5825 [] MDC={} 
2021-12-13 12:05:30,265 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-B in AddressSettings MDC={} 
2021-12-13 12:05:30,265 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-B in AddressSettings MDC={} 
2021-12-13 12:05:30,265 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@279350cf [] MDC={} 
2021-12-13 12:05:30,265 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@50732eed [] MDC={} 
2021-12-13 12:05:30,265 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-C in AddressSettings MDC={} 
2021-12-13 12:05:30,265 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-C in AddressSettings MDC={} 
2021-12-13 12:05:30,265 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@43f917b6 [] MDC={} 
2021-12-13 12:05:30,271 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) Embedded ActiveMQ started MDC={} 
2021-12-13 12:05:30,303 INFO  [io.quarkus] (Quarkus Main Thread) Quarkus 2.4.0.Final on JVM started in 6.775s. Listening on: http://localhost:8080 MDC={} 
2021-12-13 12:05:30,303 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated. MDC={} 
2021-12-13 12:05:30,303 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, config-yaml, resteasy, smallrye-context-propagation, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-amqp, swagger-ui, vertx] MDC={} 
2021-12-13 12:05:30,334 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,350 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,350 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,350 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-1 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61010 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb8b-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@2bdb332c, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@6ee38a53, true, OperationContextImpl [485968857] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-2 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61012 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb89-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@73c464e8, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@d643d7e, true, OperationContextImpl [1849626104] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-4 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61011 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb8a-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@6356bd30, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@5c2abcdb, true, OperationContextImpl [331217874] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,365 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-5 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61013 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb8c-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@70e5abb5, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@6bbf577e, true, OperationContextImpl [714381945] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,365 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-6 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61014 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb8d-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@45795daf, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@5d250485, true, OperationContextImpl [1238580376] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,365 INFO  [org.apa.act.aud.base] (Thread-3 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:61015 is creating a core session on target resource ActiveMQServerImpl::serverUUID=95746395-5c04-11ec-9cf8-0433c2e55376 [with parameters: [95bfeb8e-5c04-11ec-9cf8-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@1e691d27, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@22338e74, true, OperationContextImpl [1105142137] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 12:05:30,372 INFO  [org.apa.act.aud.base] (Thread-5 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:61013 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-C, null, 0, false, false, null]] MDC={} 
2021-12-13 12:05:30,372 INFO  [org.apa.act.aud.base] (Thread-3 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:61015 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-B, null, 0, false, false, null]] MDC={} 
2021-12-13 12:05:30,372 INFO  [org.apa.act.aud.base] (Thread-6 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:61014 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-A, null, 0, false, false, null]] MDC={} 
2021-12-13 12:05:30,387 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-C MDC={} 
2021-12-13 12:05:30,387 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-A MDC={} 
2021-12-13 12:05:30,387 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-B MDC={} 
2021-12-13 12:06:12,121 INFO  [com.min.rep.TestProducerA] (executor-thread-0) Starting 8 producer threads MDC={} 
2021-12-13 12:06:14,612 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (vert.x-eventloop-thread-0) Created worker pool named WorkerPool-A with concurrency of 1 MDC={} 
2021-12-13 12:06:17,620 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (vert.x-eventloop-thread-0) Created worker pool named WorkerPool-B with concurrency of 1 MDC={} 
2021-12-13 12:06:18,006 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (vert.x-eventloop-thread-0) Created worker pool named WorkerPool-C with concurrency of 1 MDC={} 
2021-12-13 12:07:45,680 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x MDC={} : java.lang.IllegalArgumentException: newPosition > limit: (65536 > 32768)
    at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
    at java.base/java.nio.Buffer.position(Buffer.java:293)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:262)
    at org.apache.qpid.proton.engine.impl.FrameWriterBuffer.transferTo(FrameWriterBuffer.java:215)
    at org.apache.qpid.proton.engine.impl.FrameWriter.readBytes(FrameWriter.java:73)
    at org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:376)
    at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59)
    at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.pending(SaslImpl.java:842)
    at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.pending(HandshakeSniffingTransportWrapper.java:138)
    at org.apache.qpid.proton.engine.impl.TransportImpl.pending(TransportImpl.java:1591)
    at org.apache.qpid.proton.engine.impl.TransportImpl.getOutputBuffer(TransportImpl.java:1540)
    at io.vertx.proton.impl.ProtonTransport.flush(ProtonTransport.java:273)
    at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:305)
    at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:294)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:952)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:919)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
    at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:942)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
cescoffier commented 2 years ago

Thanks, I will have a look.

cescoffier commented 2 years ago

Thanks,

So the code is not correct. When using messages, you must nack the message. The framework cannot do that, as you may have done that already:

if (!Arrays.equals(message.getPayload(), TestProducerA.TEST_DATA)) {
    return message.nack(new IllegalStateException("broken payload"));            
} else {
    return message.ack();
}

With this in B and C, I didn't get the error anymore.

belks commented 2 years ago

Hi,

I am still getting it even with the change you suggested. The only difference this time was that with the nack there were some followup errors as well (log at the end).

I would have been very surprised if the nack was fixing things, because in our actual application we do all message processing within global try-catch blocks and send a nack on any exception.

C:\develop\tools\zulu11.2.3-jdk11.0.1-win_x64\bin\java.exe "-javaagent:C:\develop\tools\JetBrains\IntelliJ IDEA 2021.2.3\lib\idea_rt.jar=56285:C:\develop\tools\JetBrains\IntelliJ IDEA 2021.2.3\bin" -Dfile.encoding=UTF-8 -classpath C:\develop\workspace\MinimalReproducer\target\classes;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy\2.4.0.Final\quarkus-resteasy-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx-http\2.4.0.Final\quarkus-vertx-http-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-security-runtime-spi\2.4.0.Final\quarkus-security-runtime-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-mutiny\2.4.0.Final\quarkus-mutiny-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx-http-dev-console-runtime-spi\2.4.0.Final\quarkus-vertx-http-dev-console-runtime-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\security\quarkus-security\1.1.4.Final\quarkus-security-1.1.4.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-web\2.14.1\smallrye-mutiny-vertx-web-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-web-common\2.14.1\smallrye-mutiny-vertx-web-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-auth-common\2.14.1\smallrye-mutiny-vertx-auth-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-bridge-common\2.14.1\smallrye-mutiny-vertx-bridge-common-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-web\4.1.5\vertx-web-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-web-common\4.1.5\vertx-web-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-auth-common\4.1.5\vertx-auth-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-bridge-common\4.1.5\vertx-bridge-common-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-core\4.1.5\vertx-core-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-resolver-dns\4.1.68.Final\netty-resolver-dns-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-dns\4.1.68.Final\netty-codec-dns-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy-server-common\2.4.0.Final\quarkus-resteasy-server-common-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-resteasy-common\2.4.0.Final\quarkus-resteasy-common-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\resteasy\resteasy-core\4.7.0.Final\resteasy-core-4.7.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\spec\javax\ws\rs\jboss-jaxrs-api_2.1_spec\2.0.1.Final\jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\spec\javax\xml\bind\jboss-jaxb-api_2.3_spec\2.0.0.Final\jboss-jaxb-api_2.3_spec-2.0.0.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\resteasy\resteasy-core-spi\4.7.0.Final\resteasy-core-spi-4.7.0.Final.jar;C:\Users\tk8es\.m2\repository\com\ibm\async\asyncutil\0.1.0\asyncutil-0.1.0.jar;C:\Users\tk8es\.m2\repository\com\sun\activation\jakarta.activation\1.2.1\jakarta.activation-1.2.1.jar;C:\Users\tk8es\.m2\repository\jakarta\validation\jakarta.validation-api\2.0.2\jakarta.validation-api-2.0.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-openapi\2.4.0.Final\quarkus-smallrye-openapi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-open-api-core\2.1.15\smallrye-open-api-core-2.1.15.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\openapi\microprofile-openapi-api\2.0.1-RC1\microprofile-openapi-api-2.0.1-RC1.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.12.5\jackson-core-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.12.5\jackson-databind-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.12.5\jackson-annotations-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\dataformat\jackson-dataformat-yaml\2.12.5\jackson-dataformat-yaml-2.12.5.jar;C:\Users\tk8es\.m2\repository\org\jboss\jandex\2.4.1.Final\jandex-2.4.1.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-core\2.4.0.Final\quarkus-core-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\jakarta\enterprise\jakarta.enterprise.cdi-api\2.0.2\jakarta.enterprise.cdi-api-2.0.2.jar;C:\Users\tk8es\.m2\repository\jakarta\el\jakarta.el-api\3.0.3\jakarta.el-api-3.0.3.jar;C:\Users\tk8es\.m2\repository\jakarta\interceptor\jakarta.interceptor-api\1.2.5\jakarta.interceptor-api-1.2.5.jar;C:\Users\tk8es\.m2\repository\jakarta\inject\jakarta.inject-api\1.0\jakarta.inject-api-1.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-ide-launcher\2.4.0.Final\quarkus-ide-launcher-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-development-mode-spi\2.4.0.Final\quarkus-development-mode-spi-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config\2.5.1\smallrye-config-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-core\2.5.1\smallrye-config-core-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-expression\1.6.0\smallrye-common-expression-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-function\1.6.0\smallrye-common-function-1.6.0.jar;C:\Users\tk8es\.m2\repository\org\jboss\logmanager\jboss-logmanager-embedded\1.0.9\jboss-logmanager-embedded-1.0.9.jar;C:\Users\tk8es\.m2\repository\org\jboss\logging\jboss-logging-annotations\2.2.1.Final\jboss-logging-annotations-2.2.1.Final.jar;C:\Users\tk8es\.m2\repository\org\jboss\threads\jboss-threads\3.4.2.Final\jboss-threads-3.4.2.Final.jar;C:\Users\tk8es\.m2\repository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;C:\Users\tk8es\.m2\repository\org\jboss\slf4j\slf4j-jboss-logmanager\1.1.0.Final\slf4j-jboss-logmanager-1.1.0.Final.jar;C:\Users\tk8es\.m2\repository\org\graalvm\sdk\graal-sdk\21.2.0\graal-sdk-21.2.0.jar;C:\Users\tk8es\.m2\repository\org\wildfly\common\wildfly-common\1.5.4.Final-format-001\wildfly-common-1.5.4.Final-format-001.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-bootstrap-runner\2.4.0.Final\quarkus-bootstrap-runner-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-io\1.6.0\smallrye-common-io-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-arc\2.4.0.Final\quarkus-arc-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\arc\arc\2.4.0.Final\arc-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\jakarta\transaction\jakarta.transaction-api\1.3.3\jakarta.transaction-api-1.3.3.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\context-propagation\microprofile-context-propagation-api\1.2\microprofile-context-propagation-api-1.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-swagger-ui\2.4.0.Final\quarkus-swagger-ui-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-config-yaml\2.4.0.Final\quarkus-config-yaml-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-source-yaml\2.5.1\smallrye-config-source-yaml-2.5.1.jar;C:\Users\tk8es\.m2\repository\org\yaml\snakeyaml\1.29\snakeyaml-1.29.jar;C:\Users\tk8es\.m2\repository\io\smallrye\config\smallrye-config-common\2.5.1\smallrye-config-common-2.5.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-classloader\1.6.0\smallrye-common-classloader-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-constraint\1.6.0\smallrye-common-constraint-1.6.0.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\config\microprofile-config-api\2.0\microprofile-config-api-2.0.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging-amqp\2.4.0.Final\quarkus-smallrye-reactive-messaging-amqp-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-amqp\3.10.1\smallrye-reactive-messaging-amqp-3.10.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-amqp-client\2.14.1\smallrye-mutiny-vertx-amqp-client-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-amqp-client\4.1.5\vertx-amqp-client-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-proton\4.1.5\vertx-proton-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-api\1.6.0\opentelemetry-api-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-context\1.6.0\opentelemetry-context-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\opentelemetry\opentelemetry-semconv\1.6.0-alpha\opentelemetry-semconv-1.6.0-alpha.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-converter-api\2.6.0\smallrye-reactive-converter-api-2.6.0.jar;C:\Users\tk8es\.m2\repository\org\reactivestreams\reactive-streams\1.0.3\reactive-streams-1.0.3.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny\1.1.2\mutiny-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-mutiny-reactive-streams-operators\2.4.0.Final\quarkus-mutiny-reactive-streams-operators-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\reactive-streams-operators\microprofile-reactive-streams-operators-api\1.0.1\microprofile-reactive-streams-operators-api-1.0.1.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\reactive-streams-operators\microprofile-reactive-streams-operators-core\1.0.1\microprofile-reactive-streams-operators-core-1.0.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny-reactive-streams-operators\1.1.2\mutiny-reactive-streams-operators-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\mutiny-smallrye-context-propagation\1.1.2\mutiny-smallrye-context-propagation-1.1.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation\1.2.2\smallrye-context-propagation-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation-api\1.2.2\smallrye-context-propagation-api-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-context-propagation-storage\1.2.2\smallrye-context-propagation-storage-1.2.2.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-context-propagation\2.4.0.Final\quarkus-smallrye-context-propagation-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging\2.4.0.Final\quarkus-smallrye-reactive-messaging-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-smallrye-reactive-messaging-kotlin\2.4.0.Final\quarkus-smallrye-reactive-messaging-kotlin-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\common\smallrye-common-annotation\1.6.0\smallrye-common-annotation-1.6.0.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-health\3.10.1\smallrye-reactive-messaging-health-3.10.1.jar;C:\Users\tk8es\.m2\repository\org\eclipse\microprofile\health\microprofile-health-api\3.1\microprofile-health-api-3.1.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-vertx\2.4.0.Final\quarkus-vertx-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-netty\2.4.0.Final\quarkus-netty-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-http2\4.1.68.Final\netty-codec-http2-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-haproxy\4.1.68.Final\netty-codec-haproxy-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-core\2.14.1\smallrye-mutiny-vertx-core-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-mutiny-vertx-runtime\2.14.1\smallrye-mutiny-vertx-runtime-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\vertx-mutiny-generator\2.14.1\vertx-mutiny-generator-2.14.1.jar;C:\Users\tk8es\.m2\repository\io\vertx\vertx-codegen\4.1.5\vertx-codegen-4.1.5.jar;C:\Users\tk8es\.m2\repository\io\smallrye\smallrye-fault-tolerance-vertx\5.2.1\smallrye-fault-tolerance-vertx-5.2.1.jar;C:\Users\tk8es\.m2\repository\io\quarkus\quarkus-jackson\2.4.0.Final\quarkus-jackson-2.4.0.Final.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.12.5\jackson-datatype-jsr310-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.12.5\jackson-datatype-jdk8-2.12.5.jar;C:\Users\tk8es\.m2\repository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.12.5\jackson-module-parameter-names-2.12.5.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-provider\3.10.1\smallrye-reactive-messaging-provider-3.10.1.jar;C:\Users\tk8es\.m2\repository\io\smallrye\reactive\smallrye-reactive-messaging-api\3.10.1\smallrye-reactive-messaging-api-3.10.1.jar;C:\Users\tk8es\.m2\repository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-amqp-protocol\2.17.0\artemis-amqp-protocol-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-selector\2.17.0\artemis-selector-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-core-client\2.17.0\artemis-core-client-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\jgroups\jgroups\3.6.13.Final\jgroups-3.6.13.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-epoll\4.1.68.Final\netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-unix-common\4.1.68.Final\netty-transport-native-unix-common-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport-native-kqueue\4.1.68.Final\netty-transport-native-kqueue-4.1.68.Final-osx-x86_64.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-http\4.1.68.Final\netty-codec-http-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-handler\4.1.68.Final\netty-handler-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-handler-proxy\4.1.68.Final\netty-handler-proxy-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec\4.1.68.Final\netty-codec-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-codec-socks\4.1.68.Final\netty-codec-socks-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-common\4.1.68.Final\netty-common-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-commons\2.17.0\artemis-commons-2.17.0.jar;C:\Users\tk8es\.m2\repository\commons-beanutils\commons-beanutils\1.9.4\commons-beanutils-1.9.4.jar;C:\Users\tk8es\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Users\tk8es\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\tk8es\.m2\repository\org\apache\geronimo\specs\geronimo-json_1.0_spec\1.0-alpha-1\geronimo-json_1.0_spec-1.0-alpha-1.jar;C:\Users\tk8es\.m2\repository\org\jboss\logging\jboss-logging\3.4.2.Final\jboss-logging-3.4.2.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-server\2.17.0\artemis-server-2.17.0.jar;C:\Users\tk8es\.m2\repository\com\google\guava\guava\30.1.1-jre\guava-30.1.1-jre.jar;C:\Users\tk8es\.m2\repository\com\google\guava\failureaccess\1.0.1\failureaccess-1.0.1.jar;C:\Users\tk8es\.m2\repository\com\google\guava\listenablefuture\9999.0-empty-to-avoid-conflict-with-guava\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar;C:\Users\tk8es\.m2\repository\com\google\code\findbugs\jsr305\3.0.2\jsr305-3.0.2.jar;C:\Users\tk8es\.m2\repository\com\google\errorprone\error_prone_annotations\2.2.0\error_prone_annotations-2.2.0.jar;C:\Users\tk8es\.m2\repository\com\google\j2objc\j2objc-annotations\1.3\j2objc-annotations-1.3.jar;C:\Users\tk8es\.m2\repository\org\jboss\logmanager\jboss-logmanager\2.1.10.Final\jboss-logmanager-2.1.10.Final.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-jdbc-store\2.17.0\artemis-jdbc-store-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-dbcp2\2.7.0\commons-dbcp2-2.7.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-pool2\2.7.0\commons-pool2-2.7.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\activemq-artemis-native\1.0.2\activemq-artemis-native-1.0.2.jar;C:\Users\tk8es\.m2\repository\org\jctools\jctools-core\2.1.2\jctools-core-2.1.2.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-configuration2\2.7\commons-configuration2-2.7.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-lang3\3.12.0\commons-lang3-3.12.0.jar;C:\Users\tk8es\.m2\repository\org\apache\commons\commons-text\1.8\commons-text-1.8.jar;C:\Users\tk8es\.m2\repository\io\micrometer\micrometer-core\1.7.4\micrometer-core-1.7.4.jar;C:\Users\tk8es\.m2\repository\org\hdrhistogram\HdrHistogram\2.1.12\HdrHistogram-2.1.12.jar;C:\Users\tk8es\.m2\repository\org\latencyutils\LatencyUtils\2.0.3\LatencyUtils-2.0.3.jar;C:\Users\tk8es\.m2\repository\commons-io\commons-io\2.11.0\commons-io-2.11.0.jar;C:\Users\tk8es\.m2\repository\org\apache\activemq\artemis-journal\2.17.0\artemis-journal-2.17.0.jar;C:\Users\tk8es\.m2\repository\org\apache\qpid\proton-j\0.33.10\proton-j-0.33.10.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-buffer\4.1.68.Final\netty-buffer-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-transport\4.1.68.Final\netty-transport-4.1.68.Final.jar;C:\Users\tk8es\.m2\repository\io\netty\netty-resolver\4.1.68.Final\netty-resolver-4.1.68.Final.jar com.minimal.reproducer.StartMe
Press [h] for more options>
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-12-13 16:31:40,109 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.log.console.json" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 16:31:40,110 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.smallrye-metrics.path" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 16:31:40,110 WARN  [io.qua.config] (Quarkus Main Thread) Unrecognized configuration key "quarkus.smallrye-health.root-path" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo MDC={} 
2021-12-13 16:31:42,176 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-A-in MDC={} 
2021-12-13 16:31:42,202 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-B-in MDC={} 
2021-12-13 16:31:42,203 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-C-in MDC={} 
2021-12-13 16:31:42,205 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-A-out MDC={} 
2021-12-13 16:31:42,215 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-B-out MDC={} 
2021-12-13 16:31:42,217 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16201: AMQP broker configured to 127.0.0.1:5672 for channel queue-C-out MDC={} 
2021-12-13 16:31:42,242 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,249 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,251 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,258 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,259 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,260 INFO  [io.sma.rea.mes.amqp] (Quarkus Main Thread) SRMSG16212: Establishing connection with AMQP broker MDC={} 
2021-12-13 16:31:42,264 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) ActiveMQ dir: C:\Users\tk8es\AppData\Local\Temp\1\activemq10576698461962543916 MDC={} 
2021-12-13 16:31:42,264 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) ActiveMQ connection properties: jms.copyMessageOnSend=false&jms.prefetchPolicy.all=1000&amqpMinLargeMessageSize=1048576&jms.useAsyncSend=true&jms.optimizeAcknowledge=true&jms.redeliveryPolicy.maximumRedeliveries=-1&compressLargeMessages=false&protocols=AMQP&jms.useCompression=false MDC={} 
2021-12-13 16:31:42,317 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) Starting embedded ActiveMQ MDC={} 
2021-12-13 16:31:42,526 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq10576698461962543916\journal,bindingsDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq10576698461962543916\bindings,largeMessagesDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq10576698461962543916\largeMessages,pagingDirectory=C:\Users\tk8es\AppData\Local\Temp\1\activemq10576698461962543916\paging) MDC={} 
2021-12-13 16:31:42,538 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221045: libaio is not available, switching the configuration into NIO MDC={} 
2021-12-13 16:31:42,558 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221013: Using NIO Journal MDC={} 
2021-12-13 16:31:42,581 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 4’269’801’472 MDC={} 
2021-12-13 16:31:42,595 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE MDC={} 
2021-12-13 16:31:42,596 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP MDC={} 
2021-12-13 16:31:42,597 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP MDC={} 
2021-12-13 16:31:42,628 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601138: User anonymous@unknown is getting notification info on target resource: null [] MDC={} 
2021-12-13 16:31:42,629 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl@4a1f0179 [] MDC={} 
2021-12-13 16:31:42,629 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601138: User anonymous@unknown is getting notification info on target resource: ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [] MDC={} 
2021-12-13 16:31:42,647 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221034: Waiting indefinitely to obtain live lock MDC={} 
2021-12-13 16:31:42,647 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221035: Live Server Obtained live lock MDC={} 
2021-12-13 16:31:42,805 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@5526c7ad [] MDC={} 
2021-12-13 16:31:42,845 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AcceptorControlImpl@5725ad76 [] MDC={} 
2021-12-13 16:31:42,881 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221020: Started NIO Acceptor at 127.0.0.1:5672 for protocols [AMQP] MDC={} 
2021-12-13 16:31:42,885 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221007: Server is now live MDC={} 
2021-12-13 16:31:42,886 INFO  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.17.0 [localhost, nodeID=c5e47204-5c29-11ec-98b6-0433c2e55376]  MDC={} 
2021-12-13 16:31:42,887 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@388f1277 [] MDC={} 
2021-12-13 16:31:42,919 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-A in AddressSettings MDC={} 
2021-12-13 16:31:42,920 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-A in AddressSettings MDC={} 
2021-12-13 16:31:42,938 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@3f9d9d6f [] MDC={} 
2021-12-13 16:31:42,940 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@24a78cee [] MDC={} 
2021-12-13 16:31:42,941 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-B in AddressSettings MDC={} 
2021-12-13 16:31:42,941 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-B in AddressSettings MDC={} 
2021-12-13 16:31:42,942 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@28d7e70e [] MDC={} 
2021-12-13 16:31:42,943 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.AddressControlImpl@3de0f2b6 [] MDC={} 
2021-12-13 16:31:42,944 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222165: No Dead Letter Address configured for queue queue-C in AddressSettings MDC={} 
2021-12-13 16:31:42,945 WARN  [org.apa.act.art.cor.server] (Quarkus Main Thread) AMQ222166: No Expiry Address configured for queue queue-C in AddressSettings MDC={} 
2021-12-13 16:31:42,946 INFO  [org.apa.act.aud.base] (Quarkus Main Thread) AMQ601019: User anonymous@unknown is getting mbean info on target resource: org.apache.activemq.artemis.core.management.impl.QueueControlImpl@2538c918 [] MDC={} 
2021-12-13 16:31:42,947 INFO  [com.min.rep.EmbeddedActiveMQServer] (Quarkus Main Thread) Embedded ActiveMQ started MDC={} 
2021-12-13 16:31:43,032 INFO  [io.quarkus] (Quarkus Main Thread) Quarkus 2.4.0.Final on JVM started in 7.155s. Listening on: http://localhost:8080 MDC={} 
2021-12-13 16:31:43,032 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated. MDC={} 
2021-12-13 16:31:43,033 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, config-yaml, resteasy, smallrye-context-propagation, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-amqp, swagger-ui, vertx] MDC={} 
2021-12-13 16:31:43,060 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,075 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,077 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,078 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,082 INFO  [org.apa.act.aud.base] (Thread-2 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56354 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63b1d89-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@2f641053, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@1e6b7915, true, OperationContextImpl [1802535748] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,082 INFO  [org.apa.act.aud.base] (Thread-3 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56355 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63b1d8a-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@7aa14a42, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@74c4a40f, true, OperationContextImpl [1637649266] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,082 INFO  [org.apa.act.aud.base] (Thread-5 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56356 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63b1d88-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@641c5edf, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@22717984, true, OperationContextImpl [1973499042] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,085 INFO  [org.apa.act.aud.base] (Thread-4 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56357 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63b92bb-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@62c0ac45, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@463b334d, true, OperationContextImpl [772152653] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,085 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,086 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established MDC={} 
2021-12-13 16:31:43,086 INFO  [org.apa.act.aud.base] (Thread-6 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56358 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63bb9cc-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@1a36d0b2, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@137ddc18, true, OperationContextImpl [1687285119] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,088 INFO  [org.apa.act.aud.base] (Thread-1 (activemq-netty-threads)) AMQ601267: User anonymous@127.0.0.1:56359 is creating a core session on target resource ActiveMQServerImpl::serverUUID=c5e47204-5c29-11ec-98b6-0433c2e55376 [with parameters: [c63c07ed-5c29-11ec-98b6-0433c2e55376, null, ****, 102400, org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection@43032a58, false, false, false, true, null, org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback@3063cae9, true, OperationContextImpl [568869819] [minimalStore=9223372036854775807, storeLineUp=0, stored=0, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=OrderedExecutor(tasks=[])], {}]] MDC={} 
2021-12-13 16:31:43,097 INFO  [org.apa.act.aud.base] (Thread-6 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:56358 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-A, null, 0, false, false, null]] MDC={} 
2021-12-13 16:31:43,097 INFO  [org.apa.act.aud.base] (Thread-1 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:56359 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-C, null, 0, false, false, null]] MDC={} 
2021-12-13 16:31:43,097 INFO  [org.apa.act.aud.base] (Thread-4 (activemq-netty-threads)) AMQ601265: User null@127.0.0.1:56357 is creating a core consumer on target resource ServerSessionImpl() [with parameters: [0, queue-B, null, 0, false, false, null]] MDC={} 
2021-12-13 16:31:43,108 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-C MDC={} 
2021-12-13 16:31:43,116 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-B MDC={} 
2021-12-13 16:31:43,118 INFO  [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16203: AMQP Receiver listening address queue-A MDC={} 
2021-12-13 16:32:27,502 INFO  [com.min.rep.TestProducerA] (executor-thread-0) Starting 8 producer threads MDC={} 
2021-12-13 16:32:30,377 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (vert.x-eventloop-thread-0) Created worker pool named WorkerPool-A with concurrency of 1 MDC={} 
2021-12-13 16:32:33,598 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (vert.x-eventloop-thread-0) Created worker pool named WorkerPool-B with concurrency of 1 MDC={} 
2021-12-13 16:32:33,970 INFO  [io.sma.rea.mes.con.WorkerPoolRegistry] (pool-1-thread-1) Created worker pool named WorkerPool-C with concurrency of 1 MDC={} 
2021-12-13 16:33:58,870 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x MDC={} : java.lang.IllegalArgumentException: newPosition > limit: (65536 > 32768)
    at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
    at java.base/java.nio.Buffer.position(Buffer.java:293)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:262)
    at org.apache.qpid.proton.engine.impl.FrameWriterBuffer.transferTo(FrameWriterBuffer.java:215)
    at org.apache.qpid.proton.engine.impl.FrameWriter.readBytes(FrameWriter.java:73)
    at org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:376)
    at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59)
    at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80)
    at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.head(SaslImpl.java:847)
    at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151)
    at org.apache.qpid.proton.engine.impl.TransportImpl.outputConsumed(TransportImpl.java:1547)
    at io.vertx.proton.impl.ProtonTransport.flush(ProtonTransport.java:279)
    at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:305)
    at io.vertx.proton.impl.ProtonTransport$IdleTimeoutCheck.handle(ProtonTransport.java:294)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:952)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:919)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
    at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:942)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-12-13 16:34:04,490 ERROR [io.sma.rea.mes.amqp] (WorkerPool-B-0) SRMSG16219: A message sent to channel `queue-B-in` has been nacked, rejecting the AMQP message and fail-stop MDC={} 
2021-12-13 16:34:04,491 ERROR [io.sma.rea.mes.amqp] (WorkerPool-B-0) SRMSG16225: Failure reported for channel `queue-B-in`, closing client MDC={} : java.lang.IllegalStateException: broken payload
    at com.minimal.reproducer.TestConsumerB.consume(TestConsumerB.java:19)
    at com.minimal.reproducer.TestConsumerB_ClientProxy.consume(TestConsumerB_ClientProxy.zig:157)
    at com.minimal.reproducer.TestConsumerB_SmallRyeMessagingInvoker_consume_88a77f5f99076f958834ad66b55b1f3dabf9589e.invoke(TestConsumerB_SmallRyeMessagingInvoker_consume_88a77f5f99076f958834ad66b55b1f3dabf9589e.zig:48)
    at io.smallrye.reactive.messaging.AbstractMediator.lambda$invokeBlocking$2(AbstractMediator.java:107)
    at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)
    at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:50)
    at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:90)
    at io.vertx.mutiny.core.WorkerExecutor$1.handle(WorkerExecutor.java:105)
    at io.vertx.mutiny.core.WorkerExecutor$1.handle(WorkerExecutor.java:103)
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
    at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-12-13 16:34:04,976 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

2021-12-13 16:34:04,978 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

2021-12-13 16:34:04,979 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

2021-12-13 16:34:05,328 WARN  [org.apa.act.art.cor.server] (Thread-5 (activemq-netty-threads)) AMQ222061: Client connection failed, clearing up resources for session c63b1d88-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:05,329 WARN  [org.apa.act.art.cor.server] (Thread-5 (activemq-netty-threads)) AMQ222107: Cleared up resources for session c63b1d88-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:05,329 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x MDC={} : java.lang.NullPointerException
    at io.vertx.amqp.impl.AmqpSenderImpl.lambda$doSend$5(AmqpSenderImpl.java:157)
    at io.vertx.proton.impl.ProtonDeliveryImpl.fireUpdate(ProtonDeliveryImpl.java:158)
    at io.vertx.proton.impl.ProtonTransport.handleSocketBuffer(ProtonTransport.java:165)
    at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:409)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$new$1(NetSocketImpl.java:97)
    at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240)
    at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$handleMessage$9(NetSocketImpl.java:390)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
    at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:389)
    at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155)
    at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:154)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-12-13 16:34:05,338 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

2021-12-13 16:34:05,385 WARN  [org.apa.act.art.cor.server] (Thread-3 (activemq-netty-threads)) AMQ222061: Client connection failed, clearing up resources for session c63b1d8a-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:05,385 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x MDC={} : java.lang.NullPointerException
    at io.vertx.amqp.impl.AmqpSenderImpl.lambda$doSend$5(AmqpSenderImpl.java:157)
    at io.vertx.proton.impl.ProtonDeliveryImpl.fireUpdate(ProtonDeliveryImpl.java:158)
    at io.vertx.proton.impl.ProtonTransport.handleSocketBuffer(ProtonTransport.java:165)
    at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:409)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$new$1(NetSocketImpl.java:97)
    at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240)
    at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$handleMessage$9(NetSocketImpl.java:390)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
    at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:389)
    at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155)
    at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:154)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-12-13 16:34:05,385 WARN  [org.apa.act.art.cor.server] (Thread-3 (activemq-netty-threads)) AMQ222107: Cleared up resources for session c63b1d8a-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:05,388 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

2021-12-13 16:34:08,097 WARN  [org.apa.act.art.cor.server] (Thread-2 (activemq-netty-threads)) AMQ222061: Client connection failed, clearing up resources for session c63b1d89-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:08,097 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x MDC={} : java.lang.NullPointerException
    at io.vertx.amqp.impl.AmqpSenderImpl.lambda$doSend$5(AmqpSenderImpl.java:157)
    at io.vertx.proton.impl.ProtonDeliveryImpl.fireUpdate(ProtonDeliveryImpl.java:158)
    at io.vertx.proton.impl.ProtonTransport.handleSocketBuffer(ProtonTransport.java:165)
    at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:409)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$new$1(NetSocketImpl.java:97)
    at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240)
    at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$handleMessage$9(NetSocketImpl.java:390)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
    at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
    at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:389)
    at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155)
    at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:154)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-12-13 16:34:08,098 WARN  [org.apa.act.art.cor.server] (Thread-2 (activemq-netty-threads)) AMQ222107: Cleared up resources for session c63b1d89-5c29-11ec-98b6-0433c2e55376 MDC={} 
2021-12-13 16:34:08,100 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception MDC={} : io.vertx.core.impl.NoStackTraceThrowable: Connection disconnected

Process finished with exit code -1
cescoffier commented 2 years ago

I finally got it reproduced. But, I don't see how we can have the race. I've added trace to all AMQP access both production, reception, ack, and nack are done on the same event loop. I also checked the idle timeout (as suggested by @gemmellr ) - same event loop.

cescoffier commented 2 years ago

I may have found something: https://github.com/vert-x3/vertx-amqp-client/blob/master/src/main/java/io/vertx/amqp/impl/AmqpSenderImpl.java#L141-L145

It accesses the message address outside of the event loop.

gemmellr commented 2 years ago

The message address is only ever returned from a field, it doesnt seem like it would influence writing to the transport buffer and updating its position, which is where the exception is being thrown, by the relatively-rarely-run idle timeout check, and in such a way that utterly smacks of there being 2 different threads operating there.

The messages themselves in general sit largely apart from the transport/engine, given they are always encoded/decoded to/from delivery bytes passed between the two areas ('messages' and 'not messages'). That passing back/forth must be single threaded but the messages once decoded/encoded it can be operated on essentially independently (until later bits like acking the delivery the bytes came from, which is again hitting the transport/engine bits and so very much single threaded)

cescoffier commented 2 years ago

Ok, I got it... It's nasty. The bug in the Vert.x AMQP client and probably there since Vert.x 4.0. The AMQP Sender uses connection.runOnContext(x -> {. However, in this case, the send is done from a "vertx.executeBlocking" construct (hidden behind the @Blocking).

The trampoline check if the current context is the attached context, and in this case, it's the case. Inside the executeBlocking, your context is still the event loop one. So it executes the action immediately... from a worker thread... BOOM....

We need to remove the trampoline, and always go back to the event loop.

cescoffier commented 2 years ago

The issue is fixed in the Vert.x AMQP client.

belks commented 2 years ago

Thank you very much :)

belks commented 2 years ago

Which release is it going to be in? I would be very happy to test it

cescoffier commented 2 years ago

Hopefully Vert.x 4.1.7 which will be followed with Reactive Messaging 3.14.0. It will be released this week.

cescoffier commented 2 years ago

It might be a 3.13.x.

cescoffier commented 2 years ago

Sorry, this is a left-over. This got fixed with the update to Vert.x 4.2.x

belks commented 2 years ago

Thank you for fixing this bug :) ...we tested it with Quarkus version 2.1.7 now, and the issue no longer happens

cescoffier commented 2 years ago

Awesome! thanks for the feedback!