search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.8k stars 2.68k forks source link

Random getting "UT000127: Response has already been sent" #15075

Closed antoniomacri closed 8 months ago

antoniomacri commented 3 years ago

Describe the bug I'm randomly getting this exception from Undertow:

java.io.IOException: java.lang.IllegalStateException: UT000127: Response has already been sent
    at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:163)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$DeferredOutputStream.write(HttpServletResponseWrapper.java:152)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$WriteOperation.doWork(HttpServletResponseWrapper.java:67)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$AsyncOperation.work(HttpServletResponseWrapper.java:41)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$DeferredOutputStream.queue(HttpServletResponseWrapper.java:217)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$DeferredOutputStream.asyncWrite(HttpServletResponseWrapper.java:179)
    at org.jboss.resteasy.util.CommitHeaderAsyncOutputStream.asyncWrite(CommitHeaderAsyncOutputStream.java:94)
    at org.jboss.resteasy.spi.AsyncOutputStream.asyncWrite(AsyncOutputStream.java:23)
    at org.jboss.resteasy.core.messagebody.AsyncBufferedMessageBodyWriter.asyncWriteTo(AsyncBufferedMessageBodyWriter.java:25)
    at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:87)
    at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.asyncProceed(AbstractWriterInterceptorContext.java:203)
    at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.getStarted(AbstractWriterInterceptorContext.java:166)
    at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.lambda$getStarted$0(ServerWriterInterceptorContext.java:73)
    at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.aroundWriteTo(ServerWriterInterceptorContext.java:93)
    at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.getStarted(ServerWriterInterceptorContext.java:73)
    at org.jboss.resteasy.core.ServerResponseWriter.lambda$writeNomapResponse$3(ServerResponseWriter.java:163)
    at org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:404)
    at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:252)
    at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:101)
    at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:74)
    at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:590)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:520)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:259)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:160)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
    at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:163)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:245)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249)
    at io.quarkus.resteasy.runtime.ResteasyFilter$ResteasyResponseWrapper.sendError(ResteasyFilter.java:65)
    at io.undertow.servlet.handlers.DefaultServlet.doGet(DefaultServlet.java:172)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
    at io.quarkus.resteasy.runtime.ResteasyFilter.doFilter(ResteasyFilter.java:28)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:63)
    at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:67)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:133)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:65)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:247)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:56)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$10$1.call(UndertowDeploymentRecorder.java:573)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
    at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:43)
    at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:114)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:290)
    at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$6$1.run(UndertowDeploymentRecorder.java:404)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1426)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
    at java.base/java.lang.Thread.run(Thread.java:834)
    at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.lang.IllegalStateException: UT000127: Response has already been sent
    at io.undertow.server.HttpServerExchange.writeBlocking(HttpServerExchange.java:1001)
    at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:155)
    ... 77 more

I cannot see any other useful info from the logs, not from the access log, neither from the same thread or similar.

Expected behavior No exception is thrown (or something better explaining why it is happening).

Actual behavior The previous exception.

To Reproduce

Sorry, I'm not able create a reproducer.

gsmet commented 3 years ago

Which version of Quarkus is it?

antoniomacri commented 3 years ago

Sorry, I forgot to specify: 1.8.3.Final.

geoand commented 3 years ago

Have you tried with Quarkus 1.11.3.Final or 1.12.0.CR1?

antoniomacri commented 3 years ago

Not yet, that is my production version. I couldn't replicate the error in my test environment.

Do you think are there any (Undertow) changes on more recent versions that could have solved this issue?

famod commented 3 years ago

Well, there is #14811/#14923, but @stuartwdouglas is the expert in this area.

stuartwdouglas commented 3 years ago

Without some form of reproducer or some more info on what causes it it will be very hard to track down. Generally this sort of thing will only happen on an error condition, where the response has already been started and then it attempts to write out an error repose.

gsmet commented 3 years ago

Do you think are there any (Undertow) changes on more recent versions that could have solved this issue?

We fixed a couple of occurrences of this error during the last months so yeah that might be already fixed in recent versions.

antoniomacri commented 3 years ago

OK, thanks, I'll try to upgrade in the next weeks (although I'd avoid to move the health endpoint).

However if I'm correct, the two mentioned issues relate to request (mine is on response), with TLS/SSL (not my case).

I just added a few logs in my app to better investigate (the error seems to be unrelated to what I see in the access log, so I don't even know which was the request uri).

antoniomacri commented 3 years ago

I noticed that most of the failing REST calls are run on a vert.x-eventloop-thread, while the others (non-failing) are on executor-thread:

Compare

02/03/2021 12:24:04,641 INFO  [io.qua.htt.access-log] (executor-thread-72552) ip=10.3.2.97 user=- method=GET requestUri="/my/path" query="-" responseCode=200 responseBytes=70359 referer="-" userAgent="-" requestId="-"

with:

02/03/2021 12:17:03,553 INFO  [io.qua.htt.access-log] (vert.x-eventloop-thread-1) ip=- user=- method=GET requestUri="/my/path" query="-" responseCode=200 responseBytes=- referer="-" userAgent="-" requestId="-"

Tipically, just after the second log I find another one with the UT000127:

02/03/2021 12:17:03,981 ERROR [org.goo.gwt.ser.CustomExceptionMapper] (executor-thread-72529) error=webApplicationException requestUri=/my/path sourceIp= errorUUID=c9e5e990-47b3-48a1-a760-230ddc185e12: java.io.IOException: java.lang.IllegalStateException: UT000127: Response has already been sent
    at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:163)
        ...

Is this somehow useful to address the issue?

stuartwdouglas commented 3 years ago

Do you have a stack trace from the event loop? The trace above was from a worker thread.

antoniomacri commented 3 years ago

I do not see any stack trace (actually no log at all) from the eventloop thread. The error is caught by an exception mapper which apparently runs on another thread: is this possible?

Do you expect something different?

antoniomacri commented 3 years ago

The error is still present on Quarkus 1.13.3.Final

What can I do to help investigate?

stuartwdouglas commented 3 years ago

Is there any chance you are using async JAX-RS, and it is timing out?

antoniomacri commented 3 years ago

Hi Stuart,

I'm using async MP-REST client in the application, but it is not involved in that call (maybe we're seeing the problem on that single endpoint because is by far the most called). That resource does something like:

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response findAll() {
        List<MyEntity> list = em.createNamedQuery("MyEntity.findAll", MyEntity.class).getResultList();
        return Response.ok().entity(list).build();
    }

This is the dependency:tree:

[INFO] it.xxx.pc:my-application-quarkus:jar:1.6.1
[INFO] +- it.xxx.pc:my-application-shared:jar:1.6.1:compile
[INFO] |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.12.1:compile
[INFO] |     +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
[INFO] |     +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.1:compile
[INFO] |     |  \- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile
[INFO] |     +- org.codehaus.woodstox:stax2-api:jar:4.2.1:compile
[INFO] |     \- com.fasterxml.woodstox:woodstox-core:jar:6.2.3:compile
[INFO] +- xxx-validation:validation:jar:69:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jboss.errai:errai-common:jar:4.2.2.Final:compile
[INFO] |  +- com.google.jsinterop:jsinterop-annotations:jar:1.0.1:compile
[INFO] |  +- de.benediktmeurer.gwt-slf4j:gwt-slf4j:jar:0.0.2:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] |  \- com.google.elemental2:elemental2-dom:jar:1.0.0-beta-1:compile
[INFO] |     +- com.google.jsinterop:base:jar:1.0.0-beta-1:compile
[INFO] |     +- com.google.elemental2:elemental2-core:jar:1.0.0-beta-1:compile
[INFO] |     \- com.google.elemental2:elemental2-promise:jar:1.0.0-beta-1:compile
[INFO] +- org.jpmml:pmml-evaluator:jar:1.5.1:compile
[INFO] |  +- org.jpmml:pmml-model:jar:1.5.1:compile
[INFO] |  |  \- org.jpmml:pmml-agent:jar:1.5.1:compile
[INFO] |  \- org.apache.commons:commons-math3:jar:3.6.1:compile (version selected from constraint [3.1,3.6.1])
[INFO] +- org.jpmml:pmml-evaluator-extension:jar:1.5.1:compile
[INFO] +- com.opencsv:opencsv:jar:5.2:test
[INFO] |  +- org.apache.commons:commons-text:jar:1.8:test
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.4:test
[INFO] |  |  +- commons-logging:commons-logging:jar:1.1.1:provided
[INFO] |  |  \- commons-collections:commons-collections:jar:3.2.2:test
[INFO] |  \- org.apache.commons:commons-collections4:jar:4.4:test
[INFO] +- org.projectlombok:lombok:jar:1.18.10:compile
[INFO] +- io.quarkus:quarkus-resteasy:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-vertx-http:jar:1.13.3.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-security-runtime-spi:jar:1.13.3.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-vertx-http-dev-console-runtime-spi:jar:1.13.3.Final:compile
[INFO] |  |  \- io.vertx:vertx-web:jar:3.9.6:compile
[INFO] |  |     +- io.vertx:vertx-web-common:jar:3.9.6:compile
[INFO] |  |     +- io.vertx:vertx-auth-common:jar:3.9.6:compile
[INFO] |  |     \- io.vertx:vertx-bridge-common:jar:3.9.6:compile
[INFO] |  \- io.quarkus:quarkus-resteasy-server-common:jar:1.13.3.Final:compile
[INFO] |     \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
[INFO] +- io.quarkus:quarkus-hibernate-orm:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-core:jar:1.13.3.Final:compile
[INFO] |  |  +- jakarta.inject:jakarta.inject-api:jar:1.0:compile
[INFO] |  |  +- io.quarkus:quarkus-ide-launcher:jar:1.13.3.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-development-mode-spi:jar:1.13.3.Final:compile
[INFO] |  |  +- io.smallrye.config:smallrye-config:jar:1.11.1:compile
[INFO] |  |  |  +- io.smallrye.common:smallrye-common-expression:jar:1.5.0:compile
[INFO] |  |  |  |  \- io.smallrye.common:smallrye-common-function:jar:1.5.0:compile
[INFO] |  |  |  \- io.smallrye.common:smallrye-common-classloader:jar:1.5.0:compile
[INFO] |  |  +- org.jboss.logmanager:jboss-logmanager-embedded:jar:1.0.9:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging-annotations:jar:2.2.0.Final:compile
[INFO] |  |  +- org.jboss.threads:jboss-threads:jar:3.2.0.Final:compile
[INFO] |  |  +- org.jboss.slf4j:slf4j-jboss-logmanager:jar:1.1.0.Final:compile
[INFO] |  |  +- org.graalvm.sdk:graal-sdk:jar:21.0.0:compile
[INFO] |  |  +- org.wildfly.common:wildfly-common:jar:1.5.4.Final-format-001:compile
[INFO] |  |  \- io.quarkus:quarkus-bootstrap-runner:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-agroal:jar:1.13.3.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-datasource:jar:1.13.3.Final:compile
[INFO] |  |  +- org.jboss.narayana.jta:narayana-jta:jar:5.10.6.Final:compile
[INFO] |  |  |  +- org.jboss:jboss-transaction-spi:jar:7.6.0.Final:compile
[INFO] |  |  |  \- org.jboss.spec.javax.resource:jboss-connector-api_1.7_spec:jar:1.0.0.Final:compile
[INFO] |  |  +- org.jboss.narayana.jts:narayana-jts-integration:jar:5.10.6.Final:compile
[INFO] |  |  +- io.agroal:agroal-api:jar:1.11:compile
[INFO] |  |  +- io.agroal:agroal-narayana:jar:1.11:compile
[INFO] |  |  +- io.agroal:agroal-pool:jar:1.11:compile
[INFO] |  |  \- io.quarkus:quarkus-credentials:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-narayana-jta:jar:1.13.3.Final:compile
[INFO] |  |  +- io.smallrye:smallrye-context-propagation-jta:jar:1.1.0:compile
[INFO] |  |  \- io.smallrye.reactive:smallrye-reactive-converter-mutiny:jar:1.4.0:compile
[INFO] |  +- org.hibernate:hibernate-core:jar:5.4.29.Final:compile
[INFO] |  |  +- antlr:antlr:jar:2.7.7:compile
[INFO] |  |  +- org.jboss:jandex:jar:2.2.3.Final:compile
[INFO] |  |  +- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  |  +- org.dom4j:dom4j:jar:2.1.3:compile
[INFO] |  |  \- org.hibernate.common:hibernate-commons-annotations:jar:5.1.2.Final:compile
[INFO] |  +- org.hibernate:hibernate-graalvm:jar:5.4.29.Final:compile
[INFO] |  +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.3-b02:compile
[INFO] |  |  +- org.glassfish.jaxb:txw2:jar:2.3.3-b02:compile
[INFO] |  |  +- com.sun.istack:istack-commons-runtime:jar:3.0.10:compile
[INFO] |  |  \- com.sun.activation:jakarta.activation:jar:1.2.1:compile
[INFO] |  +- org.jboss.spec.javax.xml.bind:jboss-jaxb-api_2.3_spec:jar:2.0.0.Final:compile
[INFO] |  +- jakarta.persistence:jakarta.persistence-api:jar:2.2.3:compile
[INFO] |  +- jakarta.transaction:jakarta.transaction-api:jar:1.3.3:compile
[INFO] |  +- org.hibernate:quarkus-local-cache:jar:0.1.0:compile
[INFO] |  |  \- com.github.ben-manes.caffeine:caffeine:jar:2.9.0:compile
[INFO] |  \- io.quarkus:quarkus-caffeine:jar:1.13.3.Final:compile
[INFO] +- io.quarkus:quarkus-quartz:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-scheduler:jar:1.13.3.Final:compile
[INFO] |  |  +- com.cronutils:cron-utils:jar:9.1.3:compile
[INFO] |  |  \- org.glassfish:jakarta.el:jar:3.0.3:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.2:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.15:compile
[INFO] +- io.quarkus:quarkus-rest-client:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-arc:jar:1.13.3.Final:compile
[INFO] |  |  \- io.quarkus.arc:arc:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-resteasy-common:jar:1.13.3.Final:compile
[INFO] |  |  \- org.jboss.resteasy:resteasy-core:jar:4.5.9.Final:compile
[INFO] |  |     +- org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_2.1_spec:jar:2.0.1.Final:compile
[INFO] |  |     +- org.jboss.resteasy:resteasy-core-spi:jar:4.5.9.Final:compile
[INFO] |  |     \- com.ibm.async:asyncutil:jar:0.1.0:compile
[INFO] |  +- org.jboss.resteasy:resteasy-client-microprofile:jar:4.5.9.Final:compile
[INFO] |  |  +- org.jboss.resteasy:resteasy-client:jar:4.5.9.Final:compile
[INFO] |  |  |  \- org.jboss.resteasy:resteasy-client-api:jar:4.5.9.Final:compile
[INFO] |  |  \- org.eclipse.microprofile.rest.client:microprofile-rest-client-api:jar:1.4.1:compile
[INFO] |  +- jakarta.interceptor:jakarta.interceptor-api:jar:1.2.5:compile
[INFO] |  +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.14:compile
[INFO] |  \- org.jboss.logging:commons-logging-jboss-logging:jar:1.0.0.Final:compile
[INFO] +- io.quarkus:quarkus-smallrye-reactive-messaging-kafka:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-kafka-client:jar:1.13.3.Final:compile
[INFO] |  +- io.smallrye.reactive:smallrye-reactive-messaging-kafka:jar:2.9.0:compile
[INFO] |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-kafka-client:jar:1.5.0:compile
[INFO] |  |  +- io.opentelemetry:opentelemetry-api:jar:1.0.1:compile
[INFO] |  |  |  \- io.opentelemetry:opentelemetry-context:jar:1.0.1:compile
[INFO] |  |  \- io.opentelemetry:opentelemetry-semconv:jar:1.0.1-alpha:compile
[INFO] |  +- io.quarkus:quarkus-jackson:jar:1.13.3.Final:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.1:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:2.7.0:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.4.5-6:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.7.1:compile
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.7.7:compile
[INFO] |  +- io.vertx:vertx-kafka-client:jar:3.9.6:compile
[INFO] |  |  \- io.vertx:vertx-core:jar:3.9.6:compile
[INFO] |  |     +- io.netty:netty-common:jar:4.1.49.Final:compile
[INFO] |  |     +- io.netty:netty-buffer:jar:4.1.49.Final:compile
[INFO] |  |     +- io.netty:netty-transport:jar:4.1.49.Final:compile
[INFO] |  |     +- io.netty:netty-handler-proxy:jar:4.1.49.Final:compile
[INFO] |  |     |  \- io.netty:netty-codec-socks:jar:4.1.49.Final:compile
[INFO] |  |     +- io.netty:netty-codec-http2:jar:4.1.49.Final:compile
[INFO] |  |     +- io.netty:netty-resolver:jar:4.1.49.Final:compile
[INFO] |  |     \- io.netty:netty-resolver-dns:jar:4.1.49.Final:compile
[INFO] |  |        \- io.netty:netty-codec-dns:jar:4.1.49.Final:compile
[INFO] |  +- io.quarkus:quarkus-smallrye-reactive-messaging:jar:1.13.3.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-mutiny-reactive-streams-operators:jar:1.13.3.Final:compile
[INFO] |  |  |  +- org.eclipse.microprofile.reactive-streams-operators:microprofile-reactive-streams-operators-api:jar:1.0.1:compile
[INFO] |  |  |  +- org.eclipse.microprofile.reactive-streams-operators:microprofile-reactive-streams-operators-core:jar:1.0.1:compile
[INFO] |  |  |  \- io.smallrye.reactive:mutiny-reactive-streams-operators:jar:0.14.0:compile
[INFO] |  |  \- io.smallrye.reactive:smallrye-reactive-messaging-health:jar:2.9.0:compile
[INFO] |  +- io.smallrye.reactive:smallrye-reactive-messaging-provider:jar:2.9.0:compile
[INFO] |  |  +- io.smallrye.reactive:smallrye-reactive-messaging-api:jar:2.9.0:compile
[INFO] |  |  \- io.reactivex.rxjava2:rxjava:jar:2.2.21:compile
[INFO] |  \- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:compile
[INFO] +- io.quarkus:quarkus-smallrye-health:jar:1.13.3.Final:compile
[INFO] |  +- io.smallrye:smallrye-health:jar:2.2.6:compile
[INFO] |  |  +- org.eclipse.microprofile.health:microprofile-health-api:jar:2.2:compile
[INFO] |  |  \- io.smallrye:smallrye-health-api:jar:2.2.6:compile
[INFO] |  +- io.smallrye:smallrye-health-provided-checks:jar:2.2.6:compile
[INFO] |  \- io.quarkus:quarkus-jsonp:jar:1.13.3.Final:compile
[INFO] |     \- org.glassfish:jakarta.json:jar:1.1.6:compile
[INFO] +- io.quarkus:quarkus-jdbc-h2:jar:1.13.3.Final:test
[INFO] |  +- com.h2database:h2:jar:1.4.197:test
[INFO] |  \- org.locationtech.jts:jts-core:jar:1.15.0:test
[INFO] +- io.quarkus:quarkus-kafka-streams:jar:1.13.3.Final:compile
[INFO] |  \- org.apache.kafka:kafka-streams:jar:2.7.0:compile
[INFO] |     +- org.apache.kafka:connect-json:jar:2.7.0:compile
[INFO] |     |  \- org.apache.kafka:connect-api:jar:2.7.0:compile
[INFO] |     \- org.rocksdb:rocksdbjni:jar:5.18.4:compile
[INFO] +- io.quarkus:quarkus-resteasy-jackson:jar:1.13.3.Final:compile
[INFO] |  \- org.jboss.resteasy:resteasy-jackson2-provider:jar:4.5.9.Final:compile
[INFO] |     +- org.jboss.resteasy:resteasy-jaxb-provider:jar:4.5.9.Final:compile
[INFO] |     +- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.1:compile
[INFO] |     |  \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.1:compile
[INFO] |     \- com.github.java-json-tools:json-patch:jar:1.13:compile
[INFO] |        +- com.github.java-json-tools:msg-simple:jar:1.2:compile
[INFO] |        |  \- com.github.java-json-tools:btf:jar:1.3:compile
[INFO] |        \- com.github.java-json-tools:jackson-coreutils:jar:2.0:compile
[INFO] +- io.quarkus:quarkus-smallrye-fault-tolerance:jar:1.13.3.Final:compile
[INFO] |  +- io.smallrye:smallrye-fault-tolerance:jar:4.3.2:compile
[INFO] |  |  +- org.eclipse.microprofile.fault-tolerance:microprofile-fault-tolerance-api:jar:2.1.1:compile
[INFO] |  |  +- io.smallrye:smallrye-fault-tolerance-api:jar:4.3.2:compile
[INFO] |  |  +- io.smallrye:smallrye-fault-tolerance-core:jar:4.3.2:compile
[INFO] |  |  +- jakarta.enterprise.concurrent:jakarta.enterprise.concurrent-api:jar:1.1.2:compile
[INFO] |  |  +- org.eclipse.microprofile.metrics:microprofile-metrics-api:jar:2.3:compile
[INFO] |  |  \- io.smallrye:smallrye-metrics-api:jar:2.4.6:compile
[INFO] |  +- jakarta.enterprise:jakarta.enterprise.cdi-api:jar:2.0.2:compile
[INFO] |  |  \- jakarta.el:jakarta.el-api:jar:3.0.3:compile
[INFO] |  \- io.smallrye:smallrye-fault-tolerance-context-propagation:jar:4.3.2:compile
[INFO] +- io.quarkus:quarkus-elytron-security-properties-file:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-elytron-security:jar:1.13.3.Final:compile
[INFO] |  |  \- io.quarkus:quarkus-elytron-security-common:jar:1.13.3.Final:compile
[INFO] |  +- org.wildfly.security:wildfly-elytron-auth-server:jar:1.15.1.Final:compile
[INFO] |  |  +- org.wildfly.security:wildfly-elytron-auth:jar:1.15.1.Final:compile
[INFO] |  |  +- org.wildfly.security:wildfly-elytron-base:jar:1.15.1.Final:compile
[INFO] |  |  +- org.wildfly.security:wildfly-elytron-credential:jar:1.15.1.Final:compile
[INFO] |  |  |  +- org.wildfly.security:wildfly-elytron-asn1:jar:1.15.1.Final:compile
[INFO] |  |  |  +- org.wildfly.security:wildfly-elytron-keystore:jar:1.15.1.Final:compile
[INFO] |  |  |  \- org.wildfly.security:wildfly-elytron-provider-util:jar:1.15.1.Final:compile
[INFO] |  |  +- org.wildfly.security:wildfly-elytron-permission:jar:1.15.1.Final:compile
[INFO] |  |  +- org.wildfly.security:wildfly-elytron-util:jar:1.15.1.Final:compile
[INFO] |  |  \- org.wildfly.security:wildfly-elytron-x500:jar:1.15.1.Final:compile
[INFO] |  +- org.wildfly.security:wildfly-elytron-password-impl:jar:1.15.1.Final:compile
[INFO] |  +- org.wildfly.security:wildfly-elytron-realm:jar:1.15.1.Final:compile
[INFO] |  +- io.quarkus:quarkus-security:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus.security:quarkus-security:jar:1.1.4.Final:compile
[INFO] |  +- jakarta.security.jacc:jakarta.security.jacc-api:jar:1.6.1:compile
[INFO] |  \- jakarta.security.auth.message:jakarta.security.auth.message-api:jar:1.1.3:compile
[INFO] +- io.quarkus:quarkus-smallrye-context-propagation:jar:1.13.3.Final:compile
[INFO] |  \- io.smallrye:smallrye-context-propagation:jar:1.1.0:compile
[INFO] |     +- io.smallrye:smallrye-context-propagation-api:jar:1.1.0:compile
[INFO] |     \- io.smallrye:smallrye-context-propagation-storage:jar:1.1.0:compile
[INFO] +- io.quarkus:quarkus-config-yaml:jar:1.13.3.Final:compile
[INFO] |  +- io.smallrye.config:smallrye-config-source-yaml:jar:1.11.1:compile
[INFO] |  |  +- org.yaml:snakeyaml:jar:1.28:compile
[INFO] |  |  +- io.smallrye.config:smallrye-config-common:jar:1.11.1:compile
[INFO] |  |  \- io.smallrye.common:smallrye-common-constraint:jar:1.5.0:compile
[INFO] |  \- org.eclipse.microprofile.config:microprofile-config-api:jar:1.4:compile
[INFO] +- io.quarkus:quarkus-jaxb:jar:1.13.3.Final:compile
[INFO] |  \- io.quarkus:quarkus-jaxp:jar:1.13.3.Final:compile
[INFO] +- io.quarkus:quarkus-resteasy-qute:jar:1.13.3.Final:compile
[INFO] |  \- io.quarkus:quarkus-qute:jar:1.13.3.Final:compile
[INFO] |     \- io.quarkus.qute:qute-core:jar:1.13.3.Final:compile
[INFO] +- io.quarkus:quarkus-vertx:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-netty:jar:1.13.3.Final:compile
[INFO] |  |  +- io.netty:netty-codec:jar:4.1.49.Final:compile
[INFO] |  |  \- io.netty:netty-handler:jar:4.1.49.Final:compile
[INFO] |  +- io.quarkus:quarkus-vertx-core:jar:1.13.3.Final:compile
[INFO] |  +- io.smallrye.common:smallrye-common-annotation:jar:1.5.0:compile
[INFO] |  +- io.quarkus:quarkus-mutiny:jar:1.13.3.Final:compile
[INFO] |  |  \- io.smallrye.reactive:mutiny-smallrye-context-propagation:jar:0.14.0:compile
[INFO] |  \- io.smallrye.reactive:smallrye-mutiny-vertx-core:jar:1.5.0:compile
[INFO] |     +- io.smallrye.reactive:smallrye-mutiny-vertx-runtime:jar:1.5.0:compile
[INFO] |     \- io.smallrye.reactive:vertx-mutiny-generator:jar:1.5.0:compile
[INFO] |        \- io.vertx:vertx-codegen:jar:3.9.6:compile
[INFO] |           \- org.mvel:mvel2:jar:2.3.1.Final:compile
[INFO] +- io.quarkus:quarkus-undertow:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus.http:quarkus-http-servlet:jar:3.1.1.Final:compile
[INFO] |  +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:compile
[INFO] |  +- io.quarkus.http:quarkus-http-vertx-backend:jar:3.1.1.Final:compile
[INFO] |  |  \- io.quarkus.http:quarkus-http-http-core:jar:3.1.1.Final:compile
[INFO] |  +- io.quarkus.http:quarkus-http-core:jar:3.1.1.Final:compile
[INFO] |  |  \- io.netty:netty-codec-http:jar:4.1.49.Final:compile
[INFO] |  \- org.eclipse.microprofile.context-propagation:microprofile-context-propagation-api:jar:1.0.1:compile
[INFO] +- io.quarkus:quarkus-oidc:jar:1.13.3.Final:compile
[INFO] |  +- io.quarkus:quarkus-oidc-common:jar:1.13.3.Final:compile
[INFO] |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-web-client:jar:1.5.0:compile
[INFO] |  |  |  +- io.vertx:vertx-web-client:jar:3.9.6:compile
[INFO] |  |  |  \- io.smallrye.reactive:smallrye-mutiny-vertx-web-common:jar:1.5.0:compile
[INFO] |  |  \- io.quarkus:quarkus-smallrye-jwt-build:jar:1.13.3.Final:compile
[INFO] |  |     \- io.smallrye:smallrye-jwt-build:jar:2.4.4:compile
[INFO] |  \- io.smallrye:smallrye-jwt:jar:2.4.4:compile
[INFO] |     +- org.eclipse.microprofile.jwt:microprofile-jwt-auth-api:jar:1.1.1:compile
[INFO] |     +- org.bitbucket.b_c:jose4j:jar:0.7.6:compile
[INFO] |     \- io.smallrye:smallrye-jwt-common:jar:2.4.4:compile
[INFO] +- com.kjetland:mbknor-jackson-jsonschema_2.12:jar:1.0.39:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.12.13:compile
[INFO] |  +- org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:jar:1.3.50:compile
[INFO] |  |  +- org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:jar:1.3.50:runtime
[INFO] |  |  |  +- org.jetbrains.kotlin:kotlin-scripting-common:jar:1.3.50:runtime
[INFO] |  |  |  |  \- org.jetbrains.kotlin:kotlin-reflect:jar:1.4.32:runtime
[INFO] |  |  |  +- org.jetbrains.kotlin:kotlin-scripting-jvm:jar:1.3.50:runtime
[INFO] |  |  |  |  \- org.jetbrains.kotlin:kotlin-script-runtime:jar:1.3.50:runtime
[INFO] |  |  |  \- org.jetbrains.kotlinx:kotlinx-coroutines-core:jar:1.1.1:runtime
[INFO] |  |  \- org.jetbrains.kotlin:kotlin-stdlib:jar:1.4.32:runtime
[INFO] |  |     +- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.4.32:runtime
[INFO] |  |     \- org.jetbrains:annotations:jar:13.0:runtime
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  \- io.github.classgraph:classgraph:jar:4.8.21:compile
[INFO] +- org.deeplearning4j:deeplearning4j-core:jar:1.0.0-beta7:compile
[INFO] |  +- org.deeplearning4j:deeplearning4j-tsne:jar:1.0.0-beta7:compile
[INFO] |  |  \- org.deeplearning4j:nearestneighbor-core:jar:1.0.0-beta7:compile
[INFO] |  +- org.deeplearning4j:deeplearning4j-datasets:jar:1.0.0-beta7:compile
[INFO] |  +- org.deeplearning4j:deeplearning4j-datavec-iterators:jar:1.0.0-beta7:compile
[INFO] |  +- org.deeplearning4j:deeplearning4j-nn:jar:1.0.0-beta7:compile
[INFO] |  |  +- org.deeplearning4j:deeplearning4j-utility-iterators:jar:1.0.0-beta7:compile
[INFO] |  |  +- org.lucee:oswego-concurrent:jar:1.3.4:compile
[INFO] |  |  +- org.nd4j:nd4j-common:jar:1.0.0-beta7:compile
[INFO] |  |  |  \- org.nd4j:guava:jar:1.0.0-beta7:compile
[INFO] |  |  \- it.unimi.dsi:fastutil:jar:6.5.7:compile
[INFO] |  +- commons-io:commons-io:jar:2.8.0:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.20:compile
[INFO] |  +- org.nd4j:nd4j-api:jar:1.0.0-beta7:compile
[INFO] |  |  +- com.jakewharton.byteunits:byteunits:jar:0.9.1:compile
[INFO] |  |  +- com.google.flatbuffers:flatbuffers-java:jar:1.10.0:compile
[INFO] |  |  +- org.nd4j:protobuf:jar:1.0.0-beta7:compile
[INFO] |  |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  |  \- net.ericaro:neoitertools:jar:1.0.0:compile
[INFO] |  +- org.nd4j:jackson:jar:1.0.0-beta7:compile
[INFO] |  +- org.datavec:datavec-api:jar:1.0.0-beta7:compile
[INFO] |  |  +- commons-codec:commons-codec:jar:1.14:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.2:compile
[INFO] |  |  +- org.freemarker:freemarker:jar:2.3.24-incubating:provided
[INFO] |  |  +- com.clearspring.analytics:stream:jar:2.9.8:compile
[INFO] |  |  +- net.sf.opencsv:opencsv:jar:2.3:compile
[INFO] |  |  \- com.tdunning:t-digest:jar:3.2:compile
[INFO] |  +- org.datavec:datavec-data-image:jar:1.0.0-beta7:compile
[INFO] |  |  +- com.github.jai-imageio:jai-imageio-core:jar:1.3.0:compile
[INFO] |  |  +- com.twelvemonkeys.imageio:imageio-jpeg:jar:3.1.1:compile
[INFO] |  |  |  +- com.twelvemonkeys.imageio:imageio-core:jar:3.1.1:compile
[INFO] |  |  |  +- com.twelvemonkeys.imageio:imageio-metadata:jar:3.1.1:compile
[INFO] |  |  |  +- com.twelvemonkeys.common:common-lang:jar:3.1.1:compile
[INFO] |  |  |  +- com.twelvemonkeys.common:common-io:jar:3.1.1:compile
[INFO] |  |  |  \- com.twelvemonkeys.common:common-image:jar:3.1.1:compile
[INFO] |  |  +- com.twelvemonkeys.imageio:imageio-tiff:jar:3.1.1:compile
[INFO] |  |  +- com.twelvemonkeys.imageio:imageio-psd:jar:3.1.1:compile
[INFO] |  |  +- com.twelvemonkeys.imageio:imageio-bmp:jar:3.1.1:compile
[INFO] |  |  +- org.bytedeco:javacv:jar:1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:ffmpeg:jar:4.2.2-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:flycapture:jar:2.13.3.31-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:libdc1394:jar:2.2.6-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:libfreenect:jar:0.5.7-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:libfreenect2:jar:0.2.0-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:librealsense:jar:1.12.4-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:librealsense2:jar:2.29.0-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:videoinput:jar:0.200-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:artoolkitplus:jar:2.3.1-1.5.3:compile
[INFO] |  |  |  +- org.bytedeco:flandmark:jar:1.07-1.5.3:compile
[INFO] |  |  |  \- org.bytedeco:tesseract:jar:4.1.1-1.5.3:compile
[INFO] |  |  +- org.bytedeco:opencv-platform:jar:4.3.0-1.5.3:compile
[INFO] |  |  \- org.bytedeco:leptonica-platform:jar:1.79.0-1.5.3:compile
[INFO] |  +- org.deeplearning4j:deeplearning4j-ui-components:jar:1.0.0-beta7:compile
[INFO] |  +- com.github.oshi:oshi-json:jar:3.4.2:compile
[INFO] |  |  \- org.glassfish:javax.json:jar:1.0.4:compile
[INFO] |  \- com.github.oshi:oshi-core:jar:3.4.2:compile
[INFO] |     +- net.java.dev.jna:jna-platform:jar:4.3.0:compile
[INFO] |     |  \- net.java.dev.jna:jna:jar:5.3.1:compile
[INFO] |     \- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] +- org.deeplearning4j:deeplearning4j-modelimport:jar:1.0.0-beta7:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] |  +- org.bytedeco:hdf5-platform:jar:1.12.0-1.5.3:compile
[INFO] |  |  \- org.bytedeco:javacpp-platform:jar:1.5.3:compile
[INFO] |  \- org.deeplearning4j:deeplearning4j-common:jar:1.0.0-beta7:compile
[INFO] +- org.nd4j:nd4j-native-platform:jar:1.0.0-beta7:compile
[INFO] |  +- org.bytedeco:openblas-platform:jar:0.3.9-1-1.5.3:compile
[INFO] |  \- org.bytedeco:mkl-platform:jar:2020.1-1.5.3:compile
[INFO] +- org.bytedeco:javacpp:jar:1.5.3:compile
[INFO] +- org.bytedeco:opencv:jar:4.3.0-1.5.3:compile
[INFO] +- org.bytedeco:leptonica:jar:1.79.0-1.5.3:compile
[INFO] +- org.bytedeco:hdf5:jar:1.12.0-1.5.3:compile
[INFO] +- org.bytedeco:openblas:jar:0.3.9-1-1.5.3:compile
[INFO] +- org.bytedeco:mkl:jar:2020.1-1.5.3:compile
[INFO] +- org.nd4j:nd4j-native:jar:1.0.0-beta7:compile
[INFO] |  \- org.nd4j:nd4j-native-api:jar:1.0.0-beta7:compile
[INFO] +- io.quarkus:quarkus-jdbc-oracle:jar:1.13.3.Final:compile
[INFO] |  \- com.oracle.database.jdbc:ojdbc8:jar:21.1.0.0:compile
[INFO] +- io.quarkus:quarkus-jdbc-postgresql:jar:1.13.3.Final:compile
[INFO] |  \- org.postgresql:postgresql:jar:42.2.20:compile
[INFO] +- com.netezza:nzjdbc:jar:1.0:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.2:compile
[INFO] +- com.jcraft:jsch:jar:0.1.55:compile
[INFO] +- io.quarkus:quarkus-junit5:jar:1.13.3.Final:test
[INFO] |  +- io.quarkus:quarkus-bootstrap-core:jar:1.13.3.Final:test
[INFO] |  |  +- io.quarkus:quarkus-bootstrap-app-model:jar:1.13.3.Final:test
[INFO] |  |  +- io.quarkus:quarkus-bootstrap-maven-resolver:jar:1.13.3.Final:test
[INFO] |  |  |  +- org.apache.maven:maven-embedder:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-settings:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-core:jar:3.6.3:test
[INFO] |  |  |  |  |  +- org.apache.maven:maven-artifact:jar:3.6.3:test
[INFO] |  |  |  |  |  \- org.codehaus.plexus:plexus-component-annotations:jar:2.1.0:test
[INFO] |  |  |  |  +- org.apache.maven:maven-plugin-api:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-model:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-model-builder:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-builder-support:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven.resolver:maven-resolver-api:jar:1.4.1:test
[INFO] |  |  |  |  +- org.apache.maven.resolver:maven-resolver-util:jar:1.4.1:test
[INFO] |  |  |  |  +- org.apache.maven.shared:maven-shared-utils:jar:3.2.1:test
[INFO] |  |  |  |  +- com.google.inject:guice:jar:no_aop:4.2.1:test
[INFO] |  |  |  |  +- org.codehaus.plexus:plexus-utils:jar:3.2.1:test
[INFO] |  |  |  |  \- org.codehaus.plexus:plexus-classworlds:jar:2.6.0:test
[INFO] |  |  |  +- org.eclipse.sisu:org.eclipse.sisu.plexus:jar:0.3.4:test
[INFO] |  |  |  +- org.apache.maven:maven-settings-builder:jar:3.6.3:test
[INFO] |  |  |  |  +- org.codehaus.plexus:plexus-interpolation:jar:1.25:test
[INFO] |  |  |  |  \- org.sonatype.plexus:plexus-sec-dispatcher:jar:1.4:test
[INFO] |  |  |  |     \- org.sonatype.plexus:plexus-cipher:jar:1.4:test
[INFO] |  |  |  +- org.apache.maven:maven-resolver-provider:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven:maven-repository-metadata:jar:3.6.3:test
[INFO] |  |  |  |  +- org.apache.maven.resolver:maven-resolver-spi:jar:1.4.1:test
[INFO] |  |  |  |  \- org.apache.maven.resolver:maven-resolver-impl:jar:1.4.1:test
[INFO] |  |  |  +- org.apache.maven.resolver:maven-resolver-connector-basic:jar:1.4.1:test
[INFO] |  |  |  +- org.apache.maven.resolver:maven-resolver-transport-wagon:jar:1.4.1:test
[INFO] |  |  |  +- org.apache.maven.wagon:wagon-http:jar:3.3.4:test
[INFO] |  |  |  |  +- org.apache.maven.wagon:wagon-http-shared:jar:3.3.4:test
[INFO] |  |  |  |  |  \- org.jsoup:jsoup:jar:1.12.1:test
[INFO] |  |  |  |  \- org.apache.maven.wagon:wagon-provider-api:jar:3.3.4:test
[INFO] |  |  |  \- org.apache.maven.wagon:wagon-file:jar:3.3.4:test
[INFO] |  |  +- io.quarkus:quarkus-bootstrap-gradle-resolver:jar:1.13.3.Final:test
[INFO] |  |  \- io.smallrye.common:smallrye-common-io:jar:1.5.0:compile
[INFO] |  +- org.eclipse.sisu:org.eclipse.sisu.inject:jar:0.3.4:test
[INFO] |  +- io.quarkus:quarkus-test-common:jar:1.13.3.Final:test
[INFO] |  |  +- io.quarkus:quarkus-core-deployment:jar:1.13.3.Final:test
[INFO] |  |  |  +- io.quarkus.gizmo:gizmo:jar:1.0.7.Final:test
[INFO] |  |  |  |  \- org.ow2.asm:asm-util:jar:9.1:test
[INFO] |  |  |  |     +- org.ow2.asm:asm-tree:jar:9.1:test
[INFO] |  |  |  |     \- org.ow2.asm:asm-analysis:jar:9.1:test
[INFO] |  |  |  +- org.ow2.asm:asm:jar:9.1:test
[INFO] |  |  |  +- io.quarkus:quarkus-class-change-agent:jar:1.13.3.Final:test
[INFO] |  |  |  +- io.quarkus:quarkus-devtools-utilities:jar:1.13.3.Final:test
[INFO] |  |  |  \- io.quarkus:quarkus-builder:jar:1.13.3.Final:test
[INFO] |  |  \- io.quarkus:quarkus-jsonp-deployment:jar:1.13.3.Final:test
[INFO] |  +- org.junit.jupiter:junit-jupiter:jar:5.7.1:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-api:jar:5.7.1:test
[INFO] |  |  |  +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] |  |  |  +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |  |  |  \- org.junit.platform:junit-platform-commons:jar:1.7.1:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-params:jar:5.7.1:test
[INFO] |  |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.7.1:test
[INFO] |  |     \- org.junit.platform:junit-platform-engine:jar:1.7.1:test
[INFO] |  +- io.quarkus:quarkus-datasource-deployment-spi:jar:1.13.3.Final:test
[INFO] |  |  \- io.quarkus:quarkus-datasource-common:jar:1.13.3.Final:compile
[INFO] |  \- com.thoughtworks.xstream:xstream:jar:1.4.16:test
[INFO] |     \- io.github.x-stream:mxparser:jar:1.2.1:test
[INFO] |        \- xmlpull:xmlpull:jar:1.1.3.1:test
[INFO] +- io.quarkus:quarkus-junit5-mockito:jar:1.13.3.Final:test
[INFO] |  \- io.quarkus:quarkus-arc-deployment:jar:1.13.3.Final:test
[INFO] |     +- io.quarkus:quarkus-vertx-http-dev-console-spi:jar:1.13.3.Final:test
[INFO] |     \- io.quarkus.arc:arc-processor:jar:1.13.3.Final:test
[INFO] +- io.quarkus:quarkus-test-security:jar:1.13.3.Final:test
[INFO] +- org.mockito:mockito-core:jar:3.3.3:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.10.5:compile
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.10.5:test
[INFO] |  \- org.objenesis:objenesis:jar:2.6:test
[INFO] +- org.assertj:assertj-core:jar:3.15.0:test
[INFO] +- io.rest-assured:rest-assured:jar:4.3.3:test
[INFO] |  +- org.codehaus.groovy:groovy:jar:3.0.7:test
[INFO] |  +- org.codehaus.groovy:groovy-xml:jar:3.0.7:test
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- org.apache.httpcomponents:httpmime:jar:4.5.13:test
[INFO] |  +- org.hamcrest:hamcrest:jar:2.1:test
[INFO] |  +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:test
[INFO] |  +- io.rest-assured:json-path:jar:4.3.3:test
[INFO] |  |  +- org.codehaus.groovy:groovy-json:jar:3.0.7:test
[INFO] |  |  \- io.rest-assured:rest-assured-common:jar:4.3.3:test
[INFO] |  \- io.rest-assured:xml-path:jar:4.3.3:test
[INFO] |     +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:compile
[INFO] |     \- com.sun.xml.bind:jaxb-impl:jar:2.3.3:test
[INFO] +- org.apache.sshd:sshd-core:jar:2.5.1:test
[INFO] |  \- org.apache.sshd:sshd-common:jar:2.5.1:test
[INFO] +- org.apache.sshd:sshd-sftp:jar:2.5.1:test
[INFO] +- com.github.marschall:memoryfilesystem:jar:2.1.0:test
[INFO] +- org.apache.kafka:kafka-streams-test-utils:jar:2.7.0:test
[INFO] +- io.smallrye.reactive:smallrye-reactive-messaging-in-memory:jar:2.9.0:test
[INFO] |  +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
[INFO] |  +- io.smallrye.reactive:smallrye-reactive-converter-api:jar:1.4.0:compile
[INFO] |  |  \- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] |  \- io.smallrye.reactive:mutiny:jar:0.14.0:compile
[INFO] +- io.debezium:debezium-core:jar:1.4.2.Final:test
[INFO] |  +- io.debezium:debezium-api:jar:1.4.2.Final:test
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.1:compile
[INFO] |  \- org.reflections:reflections:jar:0.9.12:test
[INFO] |     \- org.javassist:javassist:jar:3.26.0-GA:test
[INFO] +- io.debezium:debezium-core:test-jar:tests:1.4.2.Final:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:2.7.0:test
[INFO] |  +- org.apache.kafka:kafka-raft:jar:2.7.0:test
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.1:test
[INFO] |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.12.1:test
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.12.1:compile
[INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] |  +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.2.0:test
[INFO] |  +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.1:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.12.13:test
[INFO] |  +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.5.7:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:compile
[INFO] |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.49.Final:compile
[INFO] |  |     \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:compile
[INFO] |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] +- org.bytedeco:javacpp:jar:macosx-x86_64:1.5.3:compile
[INFO] +- org.bytedeco:opencv:jar:macosx-x86_64:4.3.0-1.5.3:compile
[INFO] +- org.bytedeco:leptonica:jar:macosx-x86_64:1.79.0-1.5.3:compile
[INFO] +- org.bytedeco:hdf5:jar:macosx-x86_64:1.12.0-1.5.3:compile
[INFO] +- org.bytedeco:openblas:jar:macosx-x86_64:0.3.9-1-1.5.3:compile
[INFO] +- org.bytedeco:mkl:jar:macosx-x86_64:2020.1-1.5.3:compile
[INFO] \- org.nd4j:nd4j-native:jar:macosx-x86_64:1.0.0-beta7:compile
Ankoul commented 3 years ago

Hi all, I am doing some stress test, so I have created a nodeJs application to send some requests per second to my quarkus application. At some point my client application starts to get "error: Error: socket hang up" in each request while my quarkus application is printing this same error for each request: 


14:23:44.2344 DEBUG [io.un.re.io] (executor-thread-504) f6a3c538-3406-45cf-994a-7e6a429c0e30 UT005013: An IOException occurred: java.io.IOException: java.lang.IllegalStateException: UT000127: Response has already been sent
    at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:249)
    at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:460)
    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:549)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:301)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:56)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$9$1.call(UndertowDeploymentRecorder.java:572)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:117)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:290)
    at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
    at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$5$1.run(UndertowDeploymentRecorder.java:398)
    at io.quarkus.runtime.CleanableExecutor$CleaningRunnable.run(CleanableExecutor.java:231)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1436)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
    at java.base/java.lang.Thread.run(Thread.java:834)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.lang.IllegalStateException: UT000127: Response has already been sent
    at io.undertow.server.HttpServerExchange.writeBlocking(HttpServerExchange.java:1001)
    at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:236)
    ... 24 more

Maybe this is an way of how to reproduce this error.

dhyegocalota commented 1 year ago

This is issue can be reproduced as simple as closing the client connection before the server completes the response! Hope that I helped somehow.

geoand commented 8 months ago

Is this still an issue with the latest versions of Quarkus?

antoniomacri commented 8 months ago

Hi @geoand, I don't know, since my application is not running in production anymore.

geoand commented 8 months ago

Understood, thanks

geoand commented 8 months ago

Closing for lack of feedback

cookier commented 6 months ago

I have encountered the same exception here as well. Fortunately, the cause was quickly identified. It was due to the Quarkus application taking too long to process requests, which exceeded the connection timeout set by Nginx, leading to the closure by Nginx and triggering this exception. The following image can reflect this situation. image