[Byte Buddy] ERROR reactor.core.publisher.ContextPropagation [jdk.internal.loader.ClassLoaders$AppClassLoader@266474c2, unnamed module @6c9f5c0d, Thread[mythread-1,5,main], loaded=false]

patpatpat123 commented 1 year ago

Expected Behavior

Run BlockHound without issue

Actual Behavior

reproducible 100%, I am seeing this issue:

[Byte Buddy] ERROR reactor.core.publisher.ContextPropagation [jdk.internal.loader.ClassLoaders$AppClassLoader@266474c2, unnamed module @6c9f5c0d, Thread[mythread-1,5,main], loaded=false]
reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$NoSuchTypeException: Cannot resolve type description for io.micrometer.context.ContextSnapshot$Scope
    at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:191)
    at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6901)
    at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.asErasure(TypePool.java:6915)
    at reactor.blockhound.shaded.net.bytebuddy.description.method.MethodDescription$AbstractBase.getDescriptor(MethodDescription.java:529)
    at reactor.blockhound.shaded.net.bytebuddy.asm.AsmVisitorWrapper$ForDeclaredMethods.wrap(AsmVisitorWrapper.java:493)
    at reactor.blockhound.shaded.net.bytebuddy.asm.AsmVisitorWrapper$Compound.wrap(AsmVisitorWrapper.java:746)
    at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithDecorationOnly$DecorationClassVisitor.visit(TypeWriter.java:5808)
    at reactor.blockhound.shaded.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:569)
    at reactor.blockhound.shaded.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:424)
    at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4014)
    at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2224)
    at reactor.blockhound.shaded.net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4057)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12225)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12160)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:11869)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12647)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12579)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12103)
    at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    at reactor.core.publisher.Mono.contextWrite(Mono.java:2338)
    at reactor.core.publisher.Mono.doOnDiscard(Mono.java:2635)
    at org.springframework.http.codec.EncoderHttpMessageWriter.write(EncoderHttpMessageWriter.java:139)
    at org.springframework.web.reactive.function.BodyInserters.lambda$write$11(BodyInserters.java:404)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at org.springframework.web.reactive.function.BodyInserters.write(BodyInserters.java:404)
    at org.springframework.web.reactive.function.BodyInserters.writeWithMessageWriters(BodyInserters.java:379)
    at org.springframework.web.reactive.function.BodyInserters.lambda$fromValue$1(BodyInserters.java:98)
    at org.springframework.web.reactive.function.client.DefaultClientRequestBuilder$BodyInserterRequest.writeTo(DefaultClientRequestBuilder.java:275)
    at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$exchange$1(ExchangeFunctions.java:102)
    at org.springframework.http.client.reactive.JettyClientHttpConnector.connect(JettyClientHttpConnector.java:125)
    at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:102)
    at org.springframework.web.reactive.function.client.DefaultWebClient$ObservationFilterFunction.filter(DefaultWebClient.java:747)
    at org.springframework.web.reactive.function.client.ExchangeFilterFunction.lambda$apply$2(ExchangeFilterFunction.java:74)
    at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultRequestBodyUriSpec.lambda$exchange$11(DefaultWebClient.java:454)
    at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:47)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4485)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:427)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258)
    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
    at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:335)
    at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:294)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341)
    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2215)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
    at reactor.core.publisher.Flux.subscribe(Flux.java:8671)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:206)
    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:345)
    at reactor.core.publisher.FluxUsing$UsingFuseableSubscriber.onNext(FluxUsing.java:353)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:440)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527)
    at reactor.core.scheduler.ExecutorScheduler$ExecutorTrackedRunnable.run(ExecutorScheduler.java:192)
    at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:84)
    at reactor.core.scheduler.SingleWorkerScheduler.execute(SingleWorkerScheduler.java:64)
    at reactor.core.scheduler.ExecutorScheduler$ExecutorSchedulerWorker.schedule(ExecutorScheduler.java:252)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.trySchedule(FluxPublishOn.java:312)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.onNext(FluxPublishOn.java:237)
    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:118)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:503)
    at reactor.core.publisher.SinkManyUnicast.drainRegular(SinkManyUnicast.java:282)
    at reactor.core.publisher.SinkManyUnicast.drain(SinkManyUnicast.java:364)
    at reactor.core.publisher.SinkManyUnicast.tryEmitNext(SinkManyUnicast.java:237)
    at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
    at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
    at reactor.kafka.receiver.internals.ConsumerEventLoop$PollEvent.run(ConsumerEventLoop.java:371)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

Steps to Reproduce

@SpringBootApplication
public class MyApplication {

    public static void main(final String[] args) {
        BlockHound.install();
        SpringApplication.run(MyApplication .class);
    }

 <dependency>
            <groupId>io.projectreactor.tools</groupId>
            <artifactId>blockhound</artifactId>
            <version>1.0.8.RELEASE</version>
        </dependency>

 <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>

note, I have as argument -XX:+AllowRedefinitionToAddDeleteMethods, not with Surefire

Possible Solution

I do not have any possible solution

Your Environment

SpringBoot 3.1.0-RC2

Reactor version(s) used:
Other relevant libraries versions (eg. netty, ...): dependencies from SpringBoot 3.1.0-RC2
JVM version (java -version): graalVM 17
OS and version (eg uname -a): reproducible with MacOS, Unix, Windows

pderop commented 1 year ago

Hi @patpatpat123 ,

Can you confirm that you do not have the context-propagation dependency in your classpath, and then, adding the dependency removes the log ?

actually, we have a pending PR in reactor-core (https://github.com/reactor/reactor-core/pull/3459) which will normally resolve this issue. The issue is in reactor-core which is triggering blockhound instrumentation on some contex-propagation types even when the optional context-propagation jar is not in the classpath (I Need to wait for team decision before being able to merge the reactor-core PR).

patpatpat123 commented 1 year ago

Hello @pderop , thank you for the quick turnaround.

Here is a list of all my dependencies:

 <dependencies>
        <dependency>
            <groupId>io.projectreactor.tools</groupId>
            <artifactId>blockhound</artifactId>
            <version>1.0.8.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.kafka</groupId>
            <artifactId>reactor-kafka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-elastic</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-reactive-httpclient</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents.core5</groupId>
            <artifactId>httpcore5</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-function-context</artifactId>
            <version>4.0.2</version>
        </dependency>
        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>7.3</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-json-serializer</artifactId>
            <version>7.3.3</version>
        </dependency>

Nothing special here, any springboot reactive kafka app would have the same.

I am not sure which "context-propagation" are you referring to.

Could you please provide the full name?

I will be more than happy to look at the entire dependency:tree and also adding it inside the pom as you suggested in your comment

pderop commented 1 year ago

normally, if you add the following in your dependencies, it should work around the problem until we release the reactor-core PR:

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>context-propagation</artifactId>
            <version>1.0.2</version>
        </dependency>

please let me know ? thanks !

patpatpat123 commented 1 year ago

You are correct! Verified. With my setup, without your dependency, the error is 100% reproducible.

Adding your dependency, error is gone!

I shall wait for reactor core PR. Closing.

Many thanks!

pderop commented 1 year ago

Duplicate of #356

reactor / BlockHound