Support for auto-instrumentation via eBPF

[NicklasWallgren]

Have DataDog any plans for adding support for auto-instrumentation via eBPF?

[felixge]

@Julio-Guerra and myself had a lot of discussions about dynamic instrumentation in the past and have explored a few directions.

Right now there are not concrete plans, but I'd be curious on what you had in mind for eBPF auto-instrumentation?

[NicklasWallgren]

We're primarily looking for support for http request and database query tracing via auto-instrumentation.

We're currently evaluating Pixie (https://px.dev/) as an alternative to Datadog, or https://github.com/keyval-dev/opentelemetry-go-instrumentation.

[felixge]

Pixie and dynamic instrumentation in general is awesome. But it's a tradeoff.

Advantages:

• Dynamically add instrumentation (logging, tracing, ...) without recompiling.
• Avoid polluting your app code with instrumentation stuff.
• Instrument runtime and library internals that would be very hard to observe otherwise.

Disadvantages:

• Overhead: uprobe's cause expensive mode switches, e.g. 3-4usec per probe hit from the testing I've done. The opentelemetry-go-instrumentation project you mentioned (and I think pixie as well?) put uprobe's in the critical path of Go's scheduler. The scheduler can usually switch between two goroutines in ~170ns, so by putting a uprobe in there you're making the scheduler 10x slower.
• Stability: If the field names or variable names that you are instrumenting change, the instrumentation will break. In the worst case this will happen silently at runtime. For new Go versions you might also have to wait until your dynamic instrumentation framework gets updated.

Anyway, we're definitely interested in dynamic instrumentation here at Datadog as well. You need to look no further than this blog post for proof of that : ).

And what I can also say is that our Database Monitoring product is making very fast advancements.

Last but not least, if your dynamic instrumentation framework is otel compatible, you should be able to use it with datadog as well.

I'll try to update this ticket as soon as we have more things to say about dynamic instrumentation for Go. But hopefully the analysis above is helpful for your decision making : )

[Julio-Guerra]

Regarding Application Security Monitoring, we already reached the point where we cannot reach the coverage we want for our HTTP monitoring without run-time instrumentation. The current plan is to start working on this topic in Q4 2022.

We will very likely start from the solution developed at Sqreen, bringing run-time instrumentation automation that can scale to large amounts of services where manual instrumentation is not conceivable.