search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.79k stars 6.52k forks source link

Exception with thrift plugin #6064

Closed furaul closed 3 years ago

furaul commented 3 years ago

Please answer these questions before submitting your issue.

Question

We are spreading skywalking in our company. Yesterday, we encountered a problem when we use thrift & thrift plugin in our project.

The error message is ERROR 2020-12-22 22:44:34:330 MessageProcess-4 ClientOutProtocolWrapper : Failed to propagating CarrierItems. java.lang.IllegalStateException: Exit span doesn't include meaningful ThreadPoolExecutor information. at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:160) at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:140) at org.apache.skywalking.apm.agent.core.context.ContextManager.inject(ContextManager.java:123) at org.apache.skywalking.apm.plugin.thrift.wrapper.ClientOutProtocolWrapper.writeFieldStop(ClientOutProtocolWrapper.java:59) at at com.xiaojukeji.code............ at org.apache.thrift.TServiceClient.sendBase$original$7XkPFApx(TServiceClient.java:71) at org.apache.thrift.TServiceClient.sendBase$original$7XkPFApx$accessor$soztpFzs(TServiceClient.java) at org.apache.thrift.TServiceClient$auxiliary$AHznHXHP.call(Unknown Source) at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at com.xiaojukeji.code............. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) We know the exception is that peer is null.

We try to debug this problem, and find that in org.apache.skywalking.apm.plugin.thrift.client.TServiceClientInterceptor

public void onConstruct(EnhancedInstance objInst, Object[] allArguments) throws NoSuchFieldException, IllegalAccessException { if (!(allArguments[1] instanceof ClientOutProtocolWrapper)) { TProtocol protocol = (TProtocol)allArguments[1]; ReflectionUtils.setValue(TServiceClient.class, objInst, "oprot_", new ClientOutProtocolWrapper(protocol)); Object dynamicField = ((EnhancedInstance)protocol.getTransport()).getSkyWalkingDynamicField();// throws Exception objInst.setSkyWalkingDynamicField(Objects.isNull(dynamicField) ? "UNKNOWN" : dynamicField); } }

Exceptions come out from the code above.

The exception message is java.lang.ClassCastException: org.apache.thrift.transport.TFramedTransport cannot be cast to org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.EnhancedInstance

It is easy to reproduce this problem.

Requirement or improvement

wu-sheng commented 3 years ago

Could you try the master branch? We recently merged a fix(#5989).

wu-sheng commented 3 years ago

And notice, our latest release is 8.3.0. The #5989 has not been released yet.

wu-sheng commented 3 years ago

No update.

1ang1ang commented 2 years ago

still have Exception use java agent:8.8.0,thrift 0.11.0, not fix this problem yet?

qiyue0421 commented 2 years ago

ERROR 2022-01-20 16:55:37:747 jaeger.RemoteReporter-QueueProcessor ClientOutProtocolWrapper : Failed to propagating CarrierItems. java.lang.IllegalStateException: Exit span doesn't include meaningful peer information. at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:168) at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:148) at org.apache.skywalking.apm.agent.core.context.ContextManager.inject(ContextManager.java:137) at org.apache.skywalking.apm.plugin.thrift.wrapper.ClientOutProtocolWrapper.writeFieldStop(ClientOutProtocolWrapper.java:59) at io.jaegertracing.thriftjava.Tag$TagStandardScheme.write(Tag.java:931) at io.jaegertracing.thriftjava.Tag$TagStandardScheme.write(Tag.java:809) at io.jaegertracing.thriftjava.Tag.write(Tag.java:709) at io.jaegertracing.thriftjava.Process$ProcessStandardScheme.write(Process.java:468) at io.jaegertracing.thriftjava.Process$ProcessStandardScheme.write(Process.java:402) at io.jaegertracing.thriftjava.Process.write(Process.java:343) at io.jaegertracing.thriftjava.Batch$BatchStandardScheme.write(Batch.java:464) at io.jaegertracing.thriftjava.Batch$BatchStandardScheme.write(Batch.java:407) at io.jaegertracing.thriftjava.Batch.write(Batch.java:344) at io.jaegertracing.agent.thrift.Agent$emitBatch_args$emitBatch_argsStandardScheme.write(Agent.java:1056) at io.jaegertracing.agent.thrift.Agent$emitBatch_args$emitBatch_argsStandardScheme.write(Agent.java:1018) at io.jaegertracing.agent.thrift.Agent$emitBatch_args.write(Agent.java:969) at org.apache.thrift.TServiceClient.sendBase$original$yadqJ0t1(TServiceClient.java:71) at org.apache.thrift.TServiceClient.sendBase$original$yadqJ0t1$accessor$q8TELQr2(TServiceClient.java) at org.apache.thrift.TServiceClient$auxiliary$IGkmczwW.call(Unknown Source) at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java) at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:64) at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:110) at io.jaegertracing.internal.reporters.RemoteReporter$FlushCommand.execute(RemoteReporter.java:157) at io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:176) at java.lang.Thread.run(Thread.java:748)

I also encountered a similar problem, for the time being can only disable the plugin

wu-sheng commented 2 years ago

@qiyue0421 You are different. From the log stack, it is clear you are using Jeager SDK, which we never tested. And why do you run 2 tracing cores at the same time? It could be a very complex scenario.

qiyue0421 commented 2 years ago

You are right, the developer feedback used jaeger sdk, maybe the previous technician testing jaeger (haha, I'm not too sure), currently using skywalking is better, sincerely thank you for your reply

pengxingjia commented 1 year ago

Which version of SkyWalking, OS, and JRE?

skywalking 9.0.0 thrift 0.14.1

Which company or project?

cisdi-info

What happened? A large number of NoopExitSpans are being generated in the application stack and not being released, resulting in a memory overflow within the application.

Object dynamicField = ((EnhancedInstance) protocol.getTransport()).getSkyWalkingDynamicField(); The exception message is java.lang.ClassCastException: org.apache.thrift.transport.TFramedTransport cannot be cast to org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.EnhancedInstance

Catching this here would eliminate the error, but I'm not sure if it would cause other issues try { Object dynamicField = ((EnhancedInstance) protocol.getTransport()).getSkyWalkingDynamicField(); objInst.setSkyWalkingDynamicField(Objects.isNull(dynamicField) ? "UNKNOWN" : dynamicField); }catch (Exception ex){ objInst.setSkyWalkingDynamicField("UNKNOWN"); }