apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.51k stars 26.43k forks source link

[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

Closed glynnwei closed 5 months ago

glynnwei commented 6 months ago

Pre-check

Search before asking

Apache Dubbo Component

Java SPI Extensions (apache/dubbo-spi-extensions)

Dubbo Version

dubbo: 3.2.11 jdk: 1.8 skywalking agent: 9.2.0

Steps to reproduce this issue

1、maven import

...
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<dubbo.version>3.2.11</dubbo.version>
<spring-boot.version>2.6.4</spring-boot.version>
...
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-registry-etcd3</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-metadata-report-etcd</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-spring-boot-observability-starter</artifactId>
    <version>3.2.11</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-metrics-api</artifactId>
    <version>3.2.11</version>
</dependency>
...

2、start javaagent

-javaagent:/skywalking-agent-9.2.0/skywalking-agent.jar
-Dskywalking.agent.service_name=test-project
-Dskywalking.collector.backend_service=127.0.0.1:11800

3、error message

java.lang.IllegalStateException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to registry 127.0.0.1:2379, cause: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
    at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:299)
    at org.apache.dubbo.registry.support.AbstractRegistry.destroy(AbstractRegistry.java:626)
    at org.apache.dubbo.registry.support.FailbackRegistry.destroy(FailbackRegistry.java:473)
    at org.apache.dubbo.registry.etcd.EtcdRegistry.destroy(EtcdRegistry.java:275)
    at org.apache.dubbo.registry.support.RegistryManager.destroyAll(RegistryManager.java:107)
    at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.destroyRegistries(DefaultApplicationDeployer.java:1372)
    at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.postDestroy(DefaultApplicationDeployer.java:1106)
    at org.apache.dubbo.rpc.model.ApplicationModel.onDestroy(ApplicationModel.java:172)
    at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
    at org.apache.dubbo.rpc.model.ApplicationModel.tryDestroy(ApplicationModel.java:276)
    at org.apache.dubbo.rpc.model.ModuleModel.onDestroy(ModuleModel.java:133)
    at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
    at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onContextClosedEvent(DubboDeployApplicationListener.java:189)
    at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:145)
    at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:52)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1058)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:172)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1021)
    at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:796)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
    at com.test.server.TestApplication.main(TestApplication.java:25)
Caused by: org.apache.dubbo.rpc.RpcException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
    at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:130)
    at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:284)
    ... 27 common frames omitted
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
    at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:481)
    at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClient.delete(JEtcdClient.java:167)
    at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:128)
    ... 28 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
    at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.lambda$delete$15(JEtcdClientWrapper.java:476)
    at org.apache.dubbo.remoting.etcd.jetcd.RetryLoops.invokeWithRetry(RetryLoops.java:57)
    at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:471)
    ... 30 common frames omitted
Caused by: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
    at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:169)
    at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:149)
    at org.apache.skywalking.apm.agent.core.context.ContextManager.inject(ContextManager.java:145)
    at org.apache.skywalking.apm.plugin.grpc.v1.client.TracingClientCall.start(TracingClientCall.java:84)
    at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:341)
    at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:315)
    at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3(ClientCalls.java:227)
    at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3$accessor$$sw$i3l8ir2(ClientCalls.java)
    at io.grpc.stub.ClientCalls$$sw$auxiliary$286ppc0.call(Unknown Source)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:83)
    at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java)
    at io.etcd.jetcd.api.KVGrpc$KVFutureStub.deleteRange(KVGrpc.java:520)
    at io.etcd.jetcd.KVImpl.lambda$delete$6(KVImpl.java:138)
    at io.etcd.jetcd.ClientConnectionManager.lambda$execute$4(ClientConnectionManager.java:349)
    at net.jodah.failsafe.Functions$1.get(Functions.java:134)
    at net.jodah.failsafe.Functions$1.get(Functions.java:129)
    at net.jodah.failsafe.Functions$2.get(Functions.java:226)
    at net.jodah.failsafe.Functions$2.get(Functions.java:216)
    at net.jodah.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:141)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

What you expected to happen

Other registries (nacos, zookeeper) can be started successfully, and skywalking data can be collected

Anything else

The /skywalk-agent-9.2.0/plugins directory is loaded properly

Are you willing to submit a pull request to fix on your own?

Code of Conduct

chickenlj commented 6 months ago

...

1.8 1.8 3.2.11 2.6.4 ... org.apache.dubbo.extensions dubbo-registry-etcd3 3.2.0 org.apache.dubbo.extensions dubbo-metadata-report-etcd 3.2.0 org.apache.dubbo dubbo-spring-boot-observability-starter 3.2.11 org.apache.dubbo dubbo-metrics-api 3.2.11 ...

Please try to add dubbo-spring-boot-starter dependency and necessary registry starter dependency. dubbo-metrics-api is not necessary once dubbo-spring-boot-starter is added.

chickenlj commented 6 months ago

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri?

glynnwei commented 6 months ago

...

1.8 1.8 3.2.11 < spring-boot.version>2.6.4 ... org.apache.dubbo.extensions dubbo-registry-etcd3 3.2.0 org.apache.dubbo.extensions dubbo-metadata-report-etcd 3.2 .0 org.apache.dubbo dubbo-spring-boot-observability-starter 3.2.11 org.apache.dubbo dubbo-metrics-api 3.2.11 ...

请尝试添加dubbo-spring-boot-starter依赖项和必要的注册表启动器依赖项。一旦添加dubbo-metrics-api就不需要了。dubbo-spring-boot-starter

<dependency>
<groupId>org.apache.dubbo</groupId>
<artifactId>dubbo-spring-boot-starter</artifactId>
<version>3.2.11</version>
</dependency>
<dependency>
<groupId>org.apache.dubbo</groupId>
<artifactId>dubbo-spring-boot-observability-starter</artifactId>
<version>3.2.11</version>
</dependency>
<dependency>
<groupId>org.apache.dubbo.extensions</groupId>
<artifactId>dubbo-registry-etcd3</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.dubbo.extensions</groupId>
<artifactId>dubbo-metadata-report-etcd</artifactId>
<version>3.2.0</version>
</dependency>

Thanks. It's still the same problem after modification

glynnwei commented 6 months ago

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri? application.yml

protocol:
name: grpc
port: 8082
registry:
address: etcd3://127.0.0.1:2379
chickenlj commented 6 months ago

More likely to be an issue of skywalking integrating with etcd.

glynnwei commented 5 months ago

I solved the problem by removing conflicting plugins:

skywalking-agent/plugins/apm-grpc-1.x-plugin-x.x.x.jar