elastic / apm

Elastic Application Performance Monitoring - resources and general issue tracking for Elastic APM.
https://www.elastic.co/apm
Apache License 2.0
384 stars 114 forks source link

[Distributed Tracing] Support Ability to Override Parent Sampling Directive with Distributed Tracing #520

Open bvader opened 3 years ago

bvader commented 3 years ago

Is your feature request related to a problem? Please describe.

As a Developer / DevOps engineer that owns "high value" centralized services I want to control / override the transaction sample_rate / instruction from upstream services that are propagated via distributed tracing headers. This is important because these centralized services are core business transactions and the upstream services my have many times the request where higher sampling is not required.

I good way to think of this as a purchase funnel, Lots of Browsing and Placing transactions and many different originating services and purchase funnels that lead to many fewer "Buying" transactions.

From the image from the doc page above. https://www.elastic.co/guide/en/apm/get-started/current/trace-sampling.html#_sampling_with_distributed_tracing dt-sampling-example

This picture exactly describes my issues. I am service C and I still need 100 sample... but also imaging 10 difference parent service all with different sampling rates calling my Service C...

In short my ability to Observe My Service C Transactions are at the Mercy of the Call / Parent Services many of which may in completely different BUs etc. which just use my published Service C endpoint

Describe the solution you'd like I would like be able to override the parent / upstream service sampling directive.

Describe alternatives you've considered Not much other than writing and HTTP Interceptor to alter the headers manually that does not seem like a good idea.

Additional context In addition some of my services may be called from 3rd parties outside my company all together, if they happen to be using Elastic APM, it seems an external entity could actually affect my internal observability.