search

opentracing-contrib / java-specialagent

Automatic instrumentation for 3rd-party libraries in Java applications with OpenTracing.
Apache License 2.0
186 stars 46 forks source link

Akka Actor integration doesn't work in a clustered environment #376

Open mayanksriv opened 4 years ago

mayanksriv commented 4 years ago

I am using sa.instrumentation.plugin.akka:actor.enable plugin to trace actor communication within my spring-boot2 app. But when my actors are distributed in a cluster there doesn't seem to be any serialization for the TracedMessage object. So basically individual nodes cannot communicate and join together in a cluster. WARN akka.cluster.Cluster(akka://foo-dispatcher-system) - Cluster Node [akka://foo-dispatcher-system@127.0.0.1:2552] - Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka://foo-dispatcher-system@127.0.0.1:2551] 2020-01-24 11:34:26 ERROR akka.remote.artery.Encoder - Failed to serialize message [io.opentracing.contrib.specialagent.rule.akka.actor.TracedMessage]. ERROR akka.remote.artery.Encoder - Failed to serialize message [io.opentracing.contrib.specialagent.rule.akka.actor.TracedMessage]. java.io.NotSerializableException: No configured serialization-bindings for class [io.opentracing.contrib.specialagent.rule.akka.actor.TracedMessage] at akka.serialization.Serialization.serializerFor(Serialization.scala:332) at akka.serialization.Serialization.findSerializerFor(Serialization.scala:307) at akka.remote.MessageSerializer$.serializeForArtery(MessageSerializer.scala:77) at akka.remote.artery.Encoder$$anon$1.onPush(Codecs.scala:136) at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523) at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:480) at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:376)

Although java-akka project has some DistibutedMessage/DistributedActor classes but it doesn't look like they are used for instrumenting distributed actors.

safris commented 4 years ago

Hi @mayanksriv, thank you for submitting this issue. We're looking into it.

malafeev commented 4 years ago

@mayanksriv java-akka doesn't have any serializer for DistributedMessage. Therefore you need to configure it.

In PR #380 I made io.opentracing.contrib.specialagent.rule.akka.actor.TracedMessage to contain only 2 fields: original message and map. It should allow to serialize it via jackson serializer for example. But as I understand you need to configure serializer for that class manually.

Another option is to intercept call akka.serialization.Serialization.serializerFor(TracedMessage.class) to get serializer of original message and use it for TracedMessage. Will it always work is not clear for me.

mayanksriv commented 4 years ago

@malafeev I ended up with my own traced proto message and used java-akka to manually instrument using tracer.inject and tracer.extract, etc. So basically just did what you've done in your changes.