Open D-D-H opened 4 months ago
In addition to the eBPF solution there is also a code generation solution called instrgen: https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrgen
cc @pdelewski
Before diving deeper, may I kindly ask if this would be similar in implementation to:
Edit: I have great respect for every project and idea, but based on previous observations, their biggest challenge often lies in the lack of a sufficient number of contributors to sustain active development. However, regardless of that, I am extremely grateful for the approach of auto-instrumentation in Go applications and I hope it can be sustained.
The idea described by this proposal is generally the same as both projects mentioned above by @jiekun. We should rather focus on what we have so far, build larger community around it and improve. I would be very happy to see more people contributing.
Thank you all for the comments.
Sorry, I didn't notice instrgen before. I'll take some time to learn about it and see how we can improve it together.
As @pdelewski said, @jiekun The principle of rewriting code of these projects you mentioned should be similar.
Thank you @D-D-H. There has been a Go-auto-instrumentation started, with two approaches led by @edeNFed and @pdelewski. I would like to see you join the Go-auto-instrumentation group, which still has a meeting on the OpenTelemetry calendar Tuesdays, 9:30am PST to discuss this proposal. If this group does not have critical mass, we recommend moving this sort of topic into the Go SIG meeting on Thursdays at 10am PST.
@jmacd Sorry for the late response and thank you for the invitation. We may not be able to participate in this meeting. How about the next meeting?
@D-D-H Feel free to join any time you want.
Hi, @jmacd @pdelewski @edeNFed I am an engineer working on observability in Alibaba Cloud, I have summarized the main difference between our approach and instrgen and the eBPF solution to the best of our knowledge and please correct me if I am wrong.
InstrGen leverages Golang's toolexec
capability to inject instrumentation code at compile time. With minor changes to the code, users could get auto instrumentation for various libraries. Additionally, it offers crucial support for asynchronous context propagation, a feature highly valued by users. While InstrGen offers several benefits, some aspects may require further evaluation for more user adoption:
Similar to InstrGen, our approach leverages compiler injection to insert instrumentation code. This approach offers several key advantages for users:
The opentelemetry-go-instrumentation project leverages eBPF uprobes for non-intrusive instrumentation of Go applications. Currently supported library includes, net/http, grpc, kafka, SQL and etc. The benefits of this approach are:
Actually we have tried the eBPF approach for a while, the considerations that we did not adopted this approach:
Please feel free to comment if you have any questions. We would like to have more discussion with the community in the upcoming SIG meeting. However, the meeting time is not quite friendly with us. I wonder is there any Asia-pacific friendly time for the meeting?
@ralf0131 From instrgen
perspective as it uses basically the same technology as your approach we can combine them to have best of both. Regarding advantages that you mentioned:
insrgen
one and would like to understand it better.
Regarding last two advantages as I said, there is no limitations from instrgen
to be on par.Currently the most important problem we are struggling is to have more people contributing to compile time instrumentation
. CC @MrAlias @pellared
@pdelewski Please see my comments in line.
@ralf0131 From
instrgen
perspective as it uses basically the same technology as your approach we can combine them to have best of both.
I agree that the two approaches share the basic idea however the implementation may varies. I think maybe we can discuss about how to combine them. However due to the difference between the two approaches I am not quite sure how to do that.
Regarding advantages that you mentioned:
- I don't think user source code modification is needed. This was initial approach, but we can eliminate that.
That will be good. How would you like to do that?
- regarding enhanced context propagation, I'm not sure how your approach is different from
insrgen
one and would like to understand it better.
Based on my understanding, assuming user has a function called foo()
, InstrGen will modified it to foo(ctx Context)
. Our approach does not modify the foo()
function to add the Context as a parameter. Instead, it adds a dedicated field within the goroutine's definition, to store the span context. When a new span is created, the span context will be stored in the goroutine's field. Subsequent retrievals of the span context from the goroutine directly access the previously stored value. To support the asynchronous propagation, during goroutine creation, a child-context can be passed along, enabling context propagation even when user does not pass the context as parameter. We will further consider baggage propagation as well in the future. The idea here is like ThreadLocal
in Java.
There is another difference regarding how the instrumented codes are injected. InstrGen analyze code from AutotelEntryPoint and build the call graph, and inject instrumentation code into functions bodies. Our approach employs a rule base policy. It customize with toolexec and analyze all the dependencies. For the dependencies that match the rule, the instrumentation code will be injected.
Regarding last two advantages as I said, there is no limitations from
instrgen
to be on par.
Could you elaborate more on how InstrGen will do that?
Currently the most important problem we are struggling is to have more people contributing to
compile time instrumentation
. CC @MrAlias @pellared
I agree with that. :)
@ralf0131 Please see my comment below.
Based on my understanding, assuming user has a function called foo(), InstrGen will modified it to foo(ctx Context). Our approach does not modify the foo() function to add the Context as a parameter. Instead, it adds a dedicated field within the goroutine's definition, to store the span context. When a new span is created, the span context will be stored in the goroutine's field. Subsequent retrievals of the span context from the goroutine directly access the previously stored value. To support the asynchronous propagation, during goroutine creation, a child-context can be passed along, enabling context propagation even when user does not pass the context as parameter. We will further consider baggage propagation as well in the future. The idea here is like ThreadLocal in Java.
There is another difference regarding how the instrumented codes are injected. InstrGen analyze code from AutotelEntryPoint and build the call graph, and inject instrumentation code into functions bodies. Our approach employs a rule base policy. It customize with toolexec and analyze all the dependencies. For the dependencies that match the rule, the instrumentation code will be injected.
Could you elaborate more on how InstrGen will do that?
Your understanding is based on what we have so far on the main branch, however there is PR https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4058 (opened almost year ago) which is more or less what you described above, so both tools follows the same techniques. Of course there still might be some implementation differences, but they are rather small from high level view.
Could you elaborate more on how InstrGen will do that?
Your understanding is based on what we have so far on the main branch, however there is PR open-telemetry/opentelemetry-go-contrib#4058 (opened almost year ago) which is more or less what you described above, so both tools follows the same techniques. Of course there still might be some implementation differences, but they are rather small from high level view.
Hi @pdelewski ,
You're right. My apologies, I hadn't noticed your pull request before and my understanding are based solely on the main branch.
I've taken a look at your implementation, and I agree that from the high level view they share the same idea. We're actually planning to open-source the latest version of our code sometime in June. This would allow us for a more in-depth discussion about combining the two approaches once we can see each other's work in detail. What do you think?
Hi @ralf0131 ,
That's great idea. Also from timeframe perspective, June sound good to me.
I saw a great idea and comments from maintainers here. As we know, providing auto-instrumentation for Go programs is difficult, but that's not the reason which cause the previous implementation being less active.
I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:
It would be great to see new projects/impls, and please also consider how we could keep them active for a long period of time.
I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:
I saw a great idea and comments from maintainers here. As we know, providing auto-instrumentation for Go programs is difficult, but that's not the reason which cause the previous implementation being less active.
I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:
- How can we incubate this (both instrgen and potentially a new implementation)? How can we increase attract more users? (External talks? Blog post?)
That's right, I think we will try to add some documentation and articles to introduce the compile time instrumentation. We also submitted proposal to the KubeCon China 2024 and hopefully we can be there to present :).
Let me share my thoughts.
It seems that most people focus on ebpf right now, no matter that it's just harder from development perspective and has tradeoffs described above. Having said that, it has one advantage that might be important for some group of people, e.g no need for recompilation or access to source code.
Another thing is that it's unfortunate that instrgen
landed in opentelemetry-go-contrib
instead of opentelemetry-go-instrumentation
so people interested in go
instrumentation usually go there and don't know about instrgen
existence.
I also tried to advertise it during KubeCon 2023 in Chicago, however that's not enough. Blogposts, articles and more people involved might help change the situation.
Let me share my thoughts.
It seems that most people focus on ebpf right now, no matter that it's just harder from development perspective and has tradeoffs described above. Having said that, it has one advantage that might be important for some group of people, e.g no need for recompilation or access to source code.
Another thing is that it's unfortunate that
instrgen
landed inopentelemetry-go-contrib
instead ofopentelemetry-go-instrumentation
so people interested ingo
instrumentation usually go there and don't know aboutinstrgen
existence.
Just be curious, may I ask why instrgen
was landed in opentelemetry-go-contrib
? Is it possible to move it to somewhere under opentelemetry-go-instrumentation
? I do think this a key reason people might be difficult to know about it. At least in the documentation, I did not find any official reference to instgen
.
I also tried to advertise it during KubeCon 2023 in Chicago, however that's not enough. Blogposts, articles and more people involved might help change the situation.
I don't think moving instrgen
to opentelemetry-go-instrumentation
is a good idea.
From a user perspective, I expect all the repositories named opentelemetry-XXX-instrumentation
will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).
Compile time instrumentation falls under the tool category and should therefore be in a different repository.
I don't think moving
instrgen
toopentelemetry-go-instrumentation
is a good idea. From a user perspective, I expect all the repositories namedopentelemetry-XXX-instrumentation
will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS). Compile time instrumentation falls under the tool category and should therefore be in a different repository.
@edeNFed Thanks for the clarification. Perhaps we should at least let users know, in the documentation, there are two approaches of instrumentation, one is the eBPF based auto instrumentation and the other is the compile time instrumentation? My suggestion will be:
Go
\--GettingStarted
\--Instrumentation
\--Automatic
\-- runtime instrumentation (eBPF based)
\-- compile time instrumentation
Looking at the documentation of Java, there is also a Spring Boot page under automatic folder, which is not a fully runtime instrumentation. To my understanding, if a Java application is compiled into a native image with GraalVM, this process is essentially a compile time instrumentation.
My second question for the eBPF-based instrumentation is, eBPF could offer a approach to instrumentation that beyond any specific programming language, why it has not been applied to other languages besides Go?
From a user perspective, I expect all the repositories named
opentelemetry-XXX-instrumentation
will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).
just fyi opentelemetry-java-instrumentation
includes both agent (runtime instrumentation) and library (build-time instrumentation)
I expect all the repositories named
opentelemetry-XXX-instrumentation
will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).
From my perspective as a user, I don't think so.
The opentelemetry-xxx-instrumentation
repository should provide automated instrumentation solutions, whether it's eBPF, Java agent, compile-time instrumentation, or something else. It different from manual instrumentation, which should be placed under the opentelemetry-xxx(-contrib)
repository.
Developers do not need to care about how Java's auto-instrumentation works. All they need to know is that they don't need to modify codes.
In short, I expect the solutions provided by the opentelemetry-xxx-instrumentation
repository to not require code changes, or at most, only minor code modifications.
The only thing shared between the eBPF instrumentation and the compiled time instrumentation is that they are both targeting Go applications.
Everything else is different, the programming language the instrumentation are written in, the contributors working on the project, tests, CI, etc.
I agree with @ralf0131 if the goal is to get more visibility into compile time instrumentation, making it more visible in the documentation is preferred in my opinion instead of mixing two unrelated projects into single repository.
The
opentelemetry-xxx-instrumentation
repository should provide automated instrumentation solutions, whether it's eBPF, Java agent, compile-time instrumentation, or something else. It different from manual instrumentation, which should be placed under theopentelemetry-xxx(-contrib)
repository.
(sort of a side conversation, but just wanted to mention that it's not quite this clear of a distinction for the Java repos at least, where the primary differentiator is that opentelemetry-java-instrumentation modules are maintained by the repo maintainers while opentelemetry-java-contrib is a distributed ownership model where individual components are maintained by component owners)
Hi all,
Update: recently we have just released our first version of compile time instrumentation on the commercial side. Now we are working on the open source our solution. It is a bit late as expected but we are working on it.
Description
The opentelemetry-go-auto-instrumentation project is an auto-instrumentation solution designed for Go applications. It empowers users to harness the capabilities of OpenTelemetry for enhanced observability without any manual modifications. Like the opentelemetry-java-instrumentation project, this solution automatically modifies code, the difference is that this all happens during the build process. The current implementation reuses the existing instrumentation for Go packages and depends on the package
dave/dst
to rewrite Go source code. The side effect of this solution is similar to the impact one would expect from manual code modifications:Benefits to the OpenTelemetry community
This project significantly lowers the barrier for Go applications to adopt OpenTelemetry. While there is an existing auto-instrumentation solution based on eBPF, it comes with certain limitations. Auto-instrumentation based on code rewriting can achieve the same effect as manual instrumentation in most scenarios and is easier to use in production.
Reasons for New Project
Drawing inspiration from the Java language, users generally prefer non-intrusive solutions (those that don't require manual code modifications). Therefore, we believe that for Go applications, this approach is likely to gain widespread acceptance among users. Making it a project of OpenTelemetry, not only ensures better maintenance but also extends the benefits to a broader user base.
Repository of Our Prototype
https://github.com/alibaba/opentelemetry-go-auto-instrumentation
Existing usage
This project is under development and has some simple demos.
Maintenance
The original contributors to this repository will continue to be involved in the project. Our current roadmap is as follows:
Licenses
Apache License 2.0
Trademarks
No Trademarks
Other notes
No response