search

reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty
https://projectreactor.io
Apache License 2.0
2.57k stars 640 forks source link

Runtime error while trying to get connection : java.lang.NoClassDefFoundError: reactor/util/concurrent/SpscLinkedArrayQueue #3307

Closed aditya1990mandrekar closed 2 months ago

aditya1990mandrekar commented 3 months ago

We have a weird issue that we saw in production couple of days back. So i have this springboot webflux application which is running in prod on Google Kubernetes engine for more than 3 months now. We faced this out of the blue bug where we got an issue connecting to one of our subsystem. To call the subsystem over http we are using WebClient. More strange thing was out of the 4 pods in prod, it appeared on only one of them.

To resolve the issue we redeployed the app and the issue seem to vanish.

Error trace:

java.lang.NoClassDefFoundError: reactor/util/concurrent/SpscLinkedArrayQueue
    at reactor.util.concurrent.Queues.lambda$unbounded$1(Queues.java:205)
    at reactor.netty.resources.DefaultPooledConnectionProvider$PendingConnectionObserver.<init>(DefaultPooledConnectionProvider.java:332)
    at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.initChannel(DefaultPooledConnectionProvider.java:545)
    at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
    at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
    at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
    at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
    at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
    at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
    at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
    at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: reactor.util.concurrent.SpscLinkedArrayQueue
    ... 22 common frames omitted

Dependencies:

[INFO] com.xxx.yyy:zzz:jar:1.0.0-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-webflux:jar:3.2.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:3.2.0:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:3.2.0:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.0:compile
[INFO] |  |  +- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:2.2:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:3.2.0:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.3:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.15.3:compile
[INFO] |  +- org.springframework:spring-web:jar:6.1.2:compile
[INFO] |  \- org.springframework:spring-webflux:jar:6.1.2:compile
[INFO] +- org.springframework.boot:spring-boot-starter-aop:jar:3.2.0:runtime
[INFO] |  +- org.springframework:spring-aop:jar:6.1.2:compile
[INFO] |  \- org.aspectj:aspectjweaver:jar:1.9.20.1:compile
[INFO] +- org.springframework.cloud:spring-cloud-gcp-starter-pubsub:jar:1.2.8.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-gcp-starter:jar:1.2.8.RELEASE:compile
[INFO] |  |  +- org.springframework.cloud:spring-cloud-gcp-core:jar:1.2.8.RELEASE:compile
[INFO] |  |  |  \- com.google.cloud:google-cloud-core:jar:2.28.0:compile
[INFO] |  |  \- org.springframework.cloud:spring-cloud-gcp-autoconfigure:jar:1.2.8.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-gcp-pubsub:jar:1.2.8.RELEASE:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:2.0.9:compile
[INFO] +- org.springframework.boot:spring-boot-starter-actuator:jar:3.2.0:runtime
[INFO] |  +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:3.2.0:compile
[INFO] |  |  \- org.springframework.boot:spring-boot-actuator:jar:3.2.0:compile
[INFO] |  +- io.micrometer:micrometer-observation:jar:1.12.0:compile
[INFO] |  |  \- io.micrometer:micrometer-commons:jar:1.12.0:compile
[INFO] |  \- io.micrometer:micrometer-jakarta9:jar:1.12.0:runtime
[INFO] +- com.github.ben-manes.caffeine:caffeine:jar:3.1.8:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:3.28.0:compile
[INFO] |  \- com.google.errorprone:error_prone_annotations:jar:2.16:compile
[INFO] +- org.springframework.boot:spring-boot-starter-cache:jar:3.2.0:compile
[INFO] |  \- org.springframework:spring-context-support:jar:6.1.2:compile
[INFO] |     \- org.springframework:spring-context:jar:6.1.2:compile
[INFO] |        \- org.springframework:spring-expression:jar:6.1.2:compile
[INFO] +- org.projectlombok:lombok:jar:1.18.30:compile
[INFO] +- com.github.dozermapper:dozer-core:jar:7.0.0:compile
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.2:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.13.0:compile
[INFO] |  +- commons-io:commons-io:jar:2.15.0:compile
[INFO] |  +- org.glassfish.jaxb:jaxb-xjc:jar:4.0.4:compile
[INFO] |  |  +- org.glassfish.jaxb:xsom:jar:4.0.4:compile
[INFO] |  |  |  \- com.sun.xml.bind.external:relaxng-datatype:jar:4.0.4:compile
[INFO] |  |  +- org.glassfish.jaxb:codemodel:jar:4.0.4:compile
[INFO] |  |  +- com.sun.xml.bind.external:rngom:jar:4.0.4:compile
[INFO] |  |  +- com.sun.xml.dtd-parser:dtd-parser:jar:1.5.0:compile
[INFO] |  |  \- com.sun.istack:istack-commons-tools:jar:4.1.2:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:2.0.9:compile
[INFO] |  \- org.objenesis:objenesis:jar:3.3:compile
[INFO] +- com.xxx.yyy:xyzs:jar:2.0.0-JDK17:compile
[INFO] |  +- com.xxx.yyy:xyz-util:jar:3.0-RC3:compile
[INFO] |  |  +- ch.qos.logback:logback-classic:jar:1.4.11:compile
[INFO] |  |  |  \- ch.qos.logback:logback-core:jar:1.4.11:compile
[INFO] |  |  +- com.google.cloud:google-cloud-logging-logback:jar:0.130.28-alpha:compile
[INFO] |  |  |  +- com.google.cloud:google-cloud-logging:jar:3.15.14:compile
[INFO] |  |  |  \- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.104.14:compile
[INFO] |  |  +- org.hibernate.orm:hibernate-core:jar:6.3.1.Final:compile
[INFO] |  |  |  +- jakarta.persistence:jakarta.persistence-api:jar:3.1.0:compile
[INFO] |  |  |  +- jakarta.transaction:jakarta.transaction-api:jar:2.0.1:compile
[INFO] |  |  |  +- org.jboss.logging:jboss-logging:jar:3.5.3.Final:runtime
[INFO] |  |  |  +- org.hibernate.common:hibernate-commons-annotations:jar:6.0.6.Final:runtime
[INFO] |  |  |  +- io.smallrye:jandex:jar:3.1.2:runtime
[INFO] |  |  |  +- com.fasterxml:classmate:jar:1.6.0:runtime
[INFO] |  |  |  +- jakarta.inject:jakarta.inject-api:jar:2.0.1:runtime
[INFO] |  |  |  \- org.antlr:antlr4-runtime:jar:4.10.1:runtime
[INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:2.0.9:compile
[INFO] |  |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |  +- org.hamcrest:hamcrest-core:jar:2.2:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.15.3:compile
[INFO] |  +- com.google.guava:guava:jar:32.1.3-jre:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.16.0:compile
[INFO] |  +- com.github.java-json-tools:jackson-coreutils:jar:1.8:compile
[INFO] |  |  \- com.github.fge:msg-simple:jar:1.1:compile
[INFO] |  |     \- com.github.fge:btf:jar:1.2:compile
[INFO] |  +- com.github.fge:json-schema-validator:jar:2.2.8:compile
[INFO] |  |  +- com.github.java-json-tools:json-schema-core:jar:1.2.8:compile
[INFO] |  |  |  +- org.mozilla:rhino:jar:1.7R4:compile
[INFO] |  |  |  +- com.github.fge:jackson-coreutils:jar:1.8:compile
[INFO] |  |  |  \- com.github.fge:uri-template:jar:0.9:compile
[INFO] |  |  +- javax.mail:mailapi:jar:1.4.3:compile
[INFO] |  |  |  \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.9.9:compile
[INFO] |  |  +- com.googlecode.libphonenumber:libphonenumber:jar:8.0.0:compile
[INFO] |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.4:compile
[INFO] |  +- net.logstash.logback:logstash-logback-encoder:jar:7.4:compile
[INFO] |  +- org.springframework:spring-core:jar:6.1.2:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:6.1.2:compile
[INFO] |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.3:compile
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:1.125.13:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.59.1:compile
[INFO] |  +- com.google.api:api-common:jar:2.21.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.29.0:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:1.20.0:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:1.20.0:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.43.3:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.107.13:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.24.0:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:2.8:compile
[INFO] |  +- com.google.api:gax:jar:2.38.0:compile
[INFO] |  +- com.google.api:gax-grpc:jar:2.38.0:compile
[INFO] |  +- io.grpc:grpc-alts:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.59.1:compile
[INFO] |  +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.2:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.59.1:compile
[INFO] |  +- io.perfmark:perfmark-api:jar:0.26.0:runtime
[INFO] |  +- io.grpc:grpc-googleapis:jar:1.59.1:runtime
[INFO] |  +- io.grpc:grpc-xds:jar:1.59.1:runtime
[INFO] |  +- io.opencensus:opencensus-proto:jar:0.2.0:runtime
[INFO] |  +- io.grpc:grpc-services:jar:1.59.1:runtime
[INFO] |  +- com.google.re2j:re2j:jar:1.6:runtime
[INFO] |  +- com.google.api:gax-httpjson:jar:2.38.0:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.10.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java-util:jar:3.25.1:compile
[INFO] |  +- org.threeten:threetenbp:jar:1.6.8:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.22.1:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-inprocess:jar:1.59.1:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.59.1:compile
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.23:runtime
[INFO] |  +- io.grpc:grpc-util:jar:1.59.1:runtime
[INFO] |  +- com.google.auto.value:auto-value-annotations:jar:1.10.4:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.43.3:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:provided
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile
[INFO] |  +- io.opencensus:opencensus-contrib-http-util:jar:0.31.1:compile
[INFO] |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] +- org.springframework:spring-messaging:jar:6.1.2:compile
[INFO] |  \- org.springframework:spring-beans:jar:6.1.2:compile
[INFO] +- io.github.resilience4j:resilience4j-spring-boot3:jar:2.1.0:compile
[INFO] |  \- io.github.resilience4j:resilience4j-spring6:jar:2.1.0:compile
[INFO] |     +- io.github.resilience4j:resilience4j-annotations:jar:2.1.0:compile
[INFO] |     +- io.github.resilience4j:resilience4j-consumer:jar:2.1.0:compile
[INFO] |     |  \- io.github.resilience4j:resilience4j-circularbuffer:jar:2.1.0:runtime
[INFO] |     \- io.github.resilience4j:resilience4j-framework-common:jar:2.1.0:compile
[INFO] +- io.github.resilience4j:resilience4j-reactor:jar:2.1.0:compile
[INFO] |  +- io.projectreactor:reactor-core:jar:3.6.0:compile
[INFO] |  |  \- org.reactivestreams:reactive-streams:jar:1.0.4:compile
[INFO] |  +- io.github.resilience4j:resilience4j-circuitbreaker:jar:2.1.0:compile
[INFO] |  |  \- io.github.resilience4j:resilience4j-core:jar:2.1.0:compile
[INFO] |  +- io.github.resilience4j:resilience4j-ratelimiter:jar:2.1.0:compile
[INFO] |  +- io.github.resilience4j:resilience4j-timelimiter:jar:2.1.0:compile
[INFO] |  +- io.github.resilience4j:resilience4j-bulkhead:jar:2.1.0:compile
[INFO] |  \- io.github.resilience4j:resilience4j-retry:jar:2.1.0:compile
[INFO] +- io.github.resilience4j:resilience4j-micrometer:jar:2.1.0:compile
[INFO] |  \- io.micrometer:micrometer-core:jar:1.12.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.12:runtime
[INFO] |     \- org.latencyutils:LatencyUtils:jar:2.0.3:runtime
[INFO] +- io.micrometer:micrometer-registry-prometheus:jar:1.12.0:compile
[INFO] |  \- io.prometheus:simpleclient_common:jar:0.16.0:compile
[INFO] |     \- io.prometheus:simpleclient:jar:0.16.0:compile
[INFO] |        +- io.prometheus:simpleclient_tracer_otel:jar:0.16.0:compile
[INFO] |        |  \- io.prometheus:simpleclient_tracer_common:jar:0.16.0:compile
[INFO] |        \- io.prometheus:simpleclient_tracer_otel_agent:jar:0.16.0:compile
[INFO] +- com.sabre.ngp.ar.starters:ngp-spring-boot-starter-logging:jar:2024.01.01:compile
[INFO] |  +- com.sabre.ngp.ar.starters:ngp-spring-boot-autoconfigure:jar:2024.01.01:compile
[INFO] |  |  +- org.jooq:joor:jar:0.9.14:compile
[INFO] |  |  +- org.owasp.encoder:encoder:jar:1.2.3:compile
[INFO] |  |  +- io.micrometer:micrometer-tracing:jar:1.2.0:compile
[INFO] |  |  |  +- io.micrometer:context-propagation:jar:1.1.0:compile
[INFO] |  |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  \- io.micrometer:micrometer-tracing-bridge-brave:jar:1.2.0:compile
[INFO] |  |     +- io.zipkin.brave:brave-context-slf4j:jar:5.16.0:compile
[INFO] |  |     +- io.zipkin.brave:brave-instrumentation-http:jar:5.16.0:compile
[INFO] |  |     \- io.zipkin.aws:brave-propagation-aws:jar:0.23.5:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-logging:jar:3.2.0:compile
[INFO] |  |  \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.21.1:compile
[INFO] |  |     \- org.apache.logging.log4j:log4j-api:jar:2.21.1:compile
[INFO] |  +- org.codehaus.janino:janino:jar:3.1.10:compile
[INFO] |  |  \- org.codehaus.janino:commons-compiler:jar:3.1.10:compile
[INFO] |  +- com.latch:logback-length-splitting-appender:jar:0.4.0:compile
[INFO] |  +- com.google.cloud:spring-cloud-gcp-starter-logging:jar:5.0.0:compile
[INFO] |  |  +- com.google.cloud:spring-cloud-gcp-starter:jar:5.0.0:compile
[INFO] |  |  |  +- com.google.cloud:spring-cloud-gcp-core:jar:5.0.0:compile
[INFO] |  |  |  \- com.google.cloud:spring-cloud-gcp-autoconfigure:jar:5.0.0:compile
[INFO] |  |  \- com.google.cloud:spring-cloud-gcp-logging:jar:5.0.0:compile
[INFO] |  |     \- ch.qos.logback.contrib:logback-json-classic:jar:0.1.5:compile
[INFO] |  |        \- ch.qos.logback.contrib:logback-json-core:jar:0.1.5:compile
[INFO] |  +- org.hobsoft.spring:spring-rest-template-logger:jar:2.0.0:compile
[INFO] |  +- com.github.chrislmy:credit-card-sanitizer:jar:1.1.2:compile
[INFO] |  +- io.zipkin.brave:brave-instrumentation-grpc:jar:5.16.0:compile
[INFO] |  |  +- io.zipkin.brave:brave-instrumentation-rpc:jar:5.16.0:compile
[INFO] |  |  \- io.zipkin.brave:brave:jar:5.16.0:compile
[INFO] |  \- com.sabre.security.ctm:audit-logging-lib:jar:1.0.1:compile
[INFO] +- com.sabre.ngp.ar.starters:ngp-spring-boot-starter-metrics:jar:2024.01.01:compile
[INFO] |  \- com.google.cloud:spring-cloud-gcp-starter-metrics:jar:5.0.0:compile
[INFO] |     \- io.micrometer:micrometer-registry-stackdriver:jar:1.12.0:compile
[INFO] |        \- com.google.cloud:google-cloud-monitoring:jar:3.32.0:compile
[INFO] |           \- com.google.api.grpc:proto-google-cloud-monitoring-v3:jar:3.32.0:compile
[INFO] +- com.sabre.ngp.ar.starters:ngp-spring-boot-starter-tracing:jar:2024.01.01:compile
[INFO] |  +- net.ttddyy.observation:datasource-micrometer-spring-boot:jar:1.0.1:compile
[INFO] |  |  \- net.ttddyy.observation:datasource-micrometer:jar:1.0.1:compile
[INFO] |  |     \- net.ttddyy:datasource-proxy:jar:1.8.1:compile
[INFO] |  \- com.google.cloud:spring-cloud-gcp-starter-trace:jar:5.0.0:compile
[INFO] |     +- org.springframework:spring-aspects:jar:6.1.2:compile
[INFO] |     +- io.zipkin.brave:brave-instrumentation-messaging:jar:5.16.0:compile
[INFO] |     +- io.zipkin.gcp:zipkin-sender-stackdriver:jar:1.0.4:compile
[INFO] |     |  +- io.zipkin.gcp:zipkin-translation-stackdriver:jar:1.0.4:compile
[INFO] |     |  |  \- com.google.api.grpc:proto-google-cloud-trace-v2:jar:2.31.0:compile
[INFO] |     |  +- io.zipkin.reporter2:zipkin-reporter:jar:2.16.3:compile
[INFO] |     |  |  \- io.zipkin.zipkin2:zipkin:jar:2.23.2:compile
[INFO] |     |  \- com.google.api.grpc:grpc-google-cloud-trace-v2:jar:2.31.0:compile
[INFO] |     +- io.zipkin.gcp:brave-propagation-stackdriver:jar:1.0.4:compile
[INFO] |     +- com.google.cloud:google-cloud-core-grpc:jar:2.28.0:compile
[INFO] |     \- io.zipkin.reporter2:zipkin-reporter-brave:jar:2.16.3:compile
[INFO] +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:3.2.0:compile
[INFO] |  \- io.projectreactor.netty:reactor-netty-http:jar:1.1.13:compile
[INFO] |     +- io.netty:netty-codec-http:jar:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-common:jar:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-buffer:jar:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-transport:jar:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-codec:jar:4.1.101.Final:compile
[INFO] |     |  \- io.netty:netty-handler:jar:4.1.101.Final:compile
[INFO] |     +- io.netty:netty-codec-http2:jar:4.1.101.Final:compile
[INFO] |     +- io.netty:netty-resolver-dns:jar:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-resolver:jar:4.1.101.Final:compile
[INFO] |     |  \- io.netty:netty-codec-dns:jar:4.1.101.Final:compile
[INFO] |     +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.101.Final:compile
[INFO] |     |  \- io.netty:netty-resolver-dns-classes-macos:jar:4.1.101.Final:compile
[INFO] |     +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.101.Final:compile
[INFO] |     |  +- io.netty:netty-transport-native-unix-common:jar:4.1.101.Final:compile
[INFO] |     |  \- io.netty:netty-transport-classes-epoll:jar:4.1.101.Final:compile
[INFO] |     \- io.projectreactor.netty:reactor-netty-core:jar:1.1.13:compile
[INFO] |        \- io.netty:netty-handler-proxy:jar:4.1.101.Final:compile
[INFO] |           \- io.netty:netty-codec-socks:jar:4.1.101.Final:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:3.2.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:3.2.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:3.2.0:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.8.0:test
[INFO] |  +- net.minidev:json-smart:jar:2.5.0:test
[INFO] |  |  \- net.minidev:accessors-smart:jar:2.5.0:test
[INFO] |  |     \- org.ow2.asm:asm:jar:9.4:test
[INFO] |  +- org.assertj:assertj-core:jar:3.24.2:test
[INFO] |  |  \- net.bytebuddy:byte-buddy:jar:1.14.10:compile
[INFO] |  +- org.awaitility:awaitility:jar:4.2.0:test
[INFO] |  +- org.hamcrest:hamcrest:jar:2.2:compile
[INFO] |  +- org.junit.jupiter:junit-jupiter:jar:5.10.1:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-params:jar:5.10.1:test
[INFO] |  |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.10.1:test
[INFO] |  +- org.mockito:mockito-junit-jupiter:jar:5.7.0:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.1:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-test:jar:6.1.2:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.9.1:test
[INFO] +- io.projectreactor:reactor-test:jar:3.6.0:test
[INFO] +- com.sun.xml.bind:jaxb-impl:jar:4.0.5:runtime
[INFO] |  \- com.sun.xml.bind:jaxb-core:jar:4.0.4:runtime
[INFO] |     \- org.eclipse.angus:angus-activation:jar:2.0.1:runtime
[INFO] +- org.mockito:mockito-inline:jar:4.5.1:test
[INFO] |  \- org.mockito:mockito-core:jar:5.7.0:compile
[INFO] |     \- net.bytebuddy:byte-buddy-agent:jar:1.14.10:compile
[INFO] +- jakarta.xml.bind:jakarta.xml.bind-api:jar:4.0.2:compile
[INFO] |  \- jakarta.activation:jakarta.activation-api:jar:2.1.2:compile
[INFO] +- org.glassfish.jaxb:jaxb-runtime:jar:4.0.5:compile
[INFO] |  \- org.glassfish.jaxb:jaxb-core:jar:4.0.4:compile
[INFO] |     +- org.glassfish.jaxb:txw2:jar:4.0.4:compile
[INFO] |     \- com.sun.istack:istack-commons-runtime:jar:4.1.2:compile
[INFO] +- org.junit.platform:junit-platform-launcher:jar:1.10.1:test
[INFO] |  +- org.junit.platform:junit-platform-engine:jar:1.10.1:test
[INFO] |  |  +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |  |  \- org.junit.platform:junit-platform-commons:jar:1.10.1:test
[INFO] |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] \- io.qameta.allure:allure-junit5:jar:2.24.0:test
[INFO]    +- io.qameta.allure:allure-junit-platform:jar:2.24.0:test
[INFO]    |  +- io.qameta.allure:allure-java-commons:jar:2.24.0:test
[INFO]    |  |  \- io.qameta.allure:allure-model:jar:2.24.0:test
[INFO]    |  \- io.qameta.allure:allure-test-filter:jar:2.24.0:test
[INFO]    \- org.junit.jupiter:junit-jupiter-api:jar:5.10.1:test

Actual Behaviour: The http call is failing Expected behaviour: http call via webclient should not fail

violetagg commented 3 months ago

@chemicL Have you seen issue like this?

chemicL commented 3 months ago

Hey. I looked into a similar case on the Micrometer Slack. I don't feel there was a conclusive outcome unfortunately, so let me just gather the resources mentioned in that thread for more transparency, but I recommend following the discussion while it's still available. The gist of it is: same type of class loading problem, Kubernetes, random occurences.

The reporter created this SO question: https://stackoverflow.com/questions/78013321/how-to-handle-the-spring-boot-classloader-problem

I managed to find a few related reports in other repositories:

To summarize, I'd look for class loaders being used and check specifically for the use of ForkJoinPool and CompletableFuture and eager use of the reactor-core APIs by some libraries, as they can run into these issues if the ordering is in conflict.

At the same time it doesn't feel this is a Spring/Reactor issue unless evidence points to us doing something fishy.

aditya1990mandrekar commented 3 months ago

One update that i missed. The issue happened again today. The common pattern between the two is, The issue occurs on the newly created pod as part of autoscaling. So looks like sporadically on the new pods, the issue occurs.

Note: we are using OpenJdk17

github-actions[bot] commented 2 months ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

github-actions[bot] commented 2 months ago

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.