petabridge / phobos-issues

Public issues and bug tracker for Phobos®
https://phobos.petabridge.com/
2 stars 1 forks source link

OpenTelemetry: fix `Activity.Current` Leaks #34

Closed Aaronontheweb closed 2 years ago

Aaronontheweb commented 2 years ago

In our internal issue tracker we've found some cases where active traces are leaking, thus breaking correlation inside our system. Happens most frequently around akka.actor.spawn and some of the new startup tracking that was added in v1.4 but ported into 2.0.

This is issue to track the status of that issue and others in the wild.

Aaronontheweb commented 2 years ago

Reproduction scenario:

  1. Parent actor /user/a receives message
  2. /user/a spawns actor `/user/a/1/ while processing message
  3. /user/a/ sends /user/a/1/ a message immediately after spawning it
  4. Parent actor completes processing of message

The parentSpanId is reported as missing for step 3 of this operation in Jaeger right now, which would indicate that the spans aren't correlated properly somewhere around the spawning stage.

Aaronontheweb commented 2 years ago

Need to add a long-running Akka.Streams scenario and validate that those spans are properly spread out over a histogram

Aaronontheweb commented 2 years ago

I have been able to verify that it is indeed the akka.actor.spawn operation that is being leaked. Have conclusive proof now.

Aaronontheweb commented 2 years ago

This has been resolved as part of Phobos 2.0.0-beta4