[Bug] DefaultFuture throw "java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom"

bert82503 commented 2 months ago

Pre-check

[X] I am sure that all the content I provide is in English.

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

Dubbo Java 3.2.10
jdk1.8.0_333

Steps to reproduce this issue

问题认识

应用启动后不久，出现了这个异常。 Shortly after the application was launched, this exception occurred.

the full exception stack:

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)
  at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
  at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.<init>(DefaultFuture.java:99)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.newFuture(DefaultFuture.java:128)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request(HeaderExchangeChannel.java:153)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.request(HeaderExchangeClient.java:109)
  at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:95)
  at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:134)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.doInvokeAndReturn(AbstractInvoker.java:248)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:191)
  at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71)
  at org.apache.dubbo.rpc.filter.RpcExceptionFilter.invoke(RpcExceptionFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
  at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:106)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:412)
  at org.apache.dubbo.rpc.cluster.support.FailfastClusterInvoker.doInvoke(FailfastClusterInvoker.java:48)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:366)
  at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0(MonitorFilter.java:108)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0$accessor$$sw$p4vq621(MonitorFilter.java)
  at org.apache.dubbo.monitor.support.MonitorFilter$$sw$auxiliary$8n9i9c3.call(Unknown Source)
  at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
  at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:62)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)

What you expected to happen

Why throw this exception?
What is the cause of this exception?
How to fix it? Not throw this exception.

Anything else

问题分析

异常调用栈，结合源代码，看着是dubbo框架引起。
当时具体发生了什么，还不知情。
The abnormal call stack, combined with the source code, appears to be caused by the Dubbo framework.
I don't know exactly what happened at that time.

dc64eb69-3c59-49cc-b262-28ef9e5c6f47

  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.<init>(DefaultFuture.java:99)

4a794b97-2ccc-4d34-ad3c-8c7d67b513ed

9868aae8-ef4d-43ca-80ad-0f89eb5fb6ee

请求重试引起的RPC异步调用？
RPC asynchronous call caused by request retry?

  at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:134)

48a84f4a-809d-449c-8e3e-ccedf76ceee0

  at org.apache.dubbo.rpc.protocol.AbstractInvoker.doInvokeAndReturn(AbstractInvoker.java:248)

3c0db96b-9511-4263-933f-36e197aa298c

  at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:191)

52ea6f11-80ee-4b28-891b-9f5f730a0faf

Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

AlbumenJ commented 2 months ago

Have you configured any javaagent?

bert82503 commented 2 months ago

Have you configured any javaagent?

skywalking-agent-9.1.0 @AlbumenJ skywalking.apm.agent apm-jdk-threadpool-plugin-9.1.0.jar

Chenjp commented 2 months ago

Does java.util.concurrent.* ever been transformed by skywalking-apm-agent?

see https://github.com/apache/skywalking/discussions/10207#discussioncomment-4502388 The simple answer is you can't. java/util/concurrent/ThreadPoolExecutor should not be instrumented.

bert82503 commented 2 months ago

Does java.util.concurrent.* ever been transformed by skywalking-apm-agent?

see apache/skywalking#10207 (comment) The simple answer is you can't. java/util/concurrent/ThreadPoolExecutor should not be instrumented.

Does java.util.concurrent.* ever been transformed by skywalking-apm-agent? Yes, we use apm-jdk-threadpool-plugin-9.1.0.jar

Okay, we can try. Thx!

bert82503 commented 1 month ago

Other application encounter this issue, which occurs when the app has just finished start run ready for K8s Liveness Probe (存活探针). And the app doesn't recover until it restarts. @Chenjp @AlbumenJ

关键根因日志

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ThreadLocalRandom.getProbe()

sun.misc.Unsafe#getInt(java.lang.Object, long) native方法

rf-guo commented 1 month ago

Does java.util.concurrent.* ever been transformed by skywalking-apm-agent? see apache/skywalking#10207 (comment) The simple answer is you can't. java/util/concurrent/ThreadPoolExecutor should not be instrumented.

Does java.util.concurrent.* ever been transformed by skywalking-apm-agent? Yes, we use apm-jdk-threadpool-plugin-9.1.0.jar

Okay, we can try. Thx!

Hi, our encounter same issue, are you has resolved?

bert82503 commented 1 month ago

Hi, our encounter same issue, are you has resolved?

SkyWalking has resolved this issue, please see apache/skywalking#12572
@rf-guo

apache / dubbo