search

kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 328 forks source link

Leaky context with Akka HTTP, Actors and Akka Streams #1024

Open steinybot opened 3 years ago

steinybot commented 3 years ago

I wrote a test which reproduces a problem that I am seeing.

The setup uses Akka HTTP which has a WebSocket route. This route handler uses the ask pattern to get an Akka Streams Source and Sink from an Actor.

What should happen is that there is a span for the route and one for the ask message but then when the source and sink are returned that should be the end of that trace.

What is actually happening is that somehow the context is leaking over into the Source. Getting a message from the WebSocket connection is replying with the current span id which is not empty.

Note that in this particular example the message is probably pulled before the client reads (although I haven't verified that) but that seems to be irrelevant to the problem. I had a much more complicated example where it would reply only when a message was sent and the behaviour is the same.

Interestingly if I trying and switch it out for a Flow and use upgrade.handleMessages the problem goes away. There seems to be something different about using a Source.

SimunKaracic commented 3 years ago

Good job investigating and writing a detailed bug report! You also found a workaround :D Hopefully, it will be enough for now.

The problem with Streams is that they're a generic thing, and anything can happen between a Source and a Sink. That means it's not clear exactly where we should hook in to start/stop measuring things. For now, it's recommended to stay away from them, replace them with Flows, or wrap your routes in ServerFlowWrappers. At some point (soon, hopefully,) we'll have to dig into them and figure out how to instrument them, but for now these workarounds will have todo.

I'll update the docs this week to reflect this.