search

petabridge / phobos-issues

Public issues and bug tracker for Phobos®
https://phobos.petabridge.com/
2 stars 1 forks source link

Unable to filter out noise from trace open telemetry #49

Closed TerryDieckmann closed 2 years ago

TerryDieckmann commented 2 years ago

I am currently investigating how we can best use OTel and Phobos in our software solution. During the implementation I can't really figure out how to filter out the messages below without adding an endless list of separate message filters.

The trace-actor-lifecycle option is disabled, but there are still plenty of internal akka (unnecessary) messages.

image

Using .NET 6, Akka.NET Cluster, Open Telemetry and Phobos.

Aaronontheweb commented 2 years ago

Thanks Terry - we'll take a look at this.

TerryDieckmann commented 2 years ago

Thanks Aaron, It might be good to know that I use Phobos.Actor.Cluster 2.0.0-beta3. I've tried filtering out the bulk of (to me) unnecessary messages, but when I do that, half of the trace is no longer included because it is a child of akka.msg.recv GetShardHome.

Aaronontheweb commented 2 years ago

We have another issue related to trace filtering on Phobos 1.4 with #46 - so let's just say that this has been clearly identified as a "user experience" problem across all use cases ;)

We learned some lessons on how to handle filtering more robustly when we implemented type filters for security purposes inside Hyperion https://github.com/akkadotnet/akka.net/discussions/5329#discussioncomment-1560737

I'll work on improving the upfront filtering and also add an example for using an OTel Processor on the back-end.

Aaronontheweb commented 2 years ago

Wrote up our plans for addressing this on a related issue here: https://github.com/petabridge/phobos-issues/issues/46#issuecomment-1070908283 - would love your feedback on it @TerryDieckmann

Aaronontheweb commented 2 years ago

Hi @TerryDieckmann - we have pushed new Phobos 2.0 binaries that have an API change that should nip this problem in the bud.

Please see the release notes here and follow the configuration instructions - I think this should help quite a bit: https://phobos.petabridge.com/articles/releases/RELEASE_NOTES.html#200-beta4httpssdkbincompublisherpetabridgeproductphobospackagesphobosactorversions200-beta4-and-150-beta1httpssdkbincompublisherpetabridgeproductphobospackagesphobosactorversions150-beta1-march-17th-2022

TerryDieckmann commented 2 years ago

I will have a look, thanks!

TerryDieckmann commented 2 years ago

I've implemented my own tracer filter and it works great! Traces are complete and not full of messages we don't really need. Thank you, Aaron!

Aaronontheweb commented 2 years ago

Glad to hear it!