Terminology Suggestion: Change "Manual Instrumentation" to "Code based Instrumentation" in docs

[martinjt]

This is more of a wider suggestion based on a lot of conversations I've been having recently about the ways people instrument their applications.

The general feeling I've been hearing is that "If I'm using Manual Instrumentation I'm doing it wrong". For context, this is from engineering conferences primarily focused on JS/TS and .NET, I understand that languages like Java may put more focus on AOP style techniques that may make Manual actually be more of an anti-pattern.

Automatic instrumentation, using an external application to generate/setup tracing/metrics/logs is not a bad thing, I'm not at all saying that, I'm also not saying that Manual is the one true way. I'm saying that the terminology is making people I speak to a little confused about the "right" approach.

I talk about 3 different avenues to instrumentation with Otel.

• Auto/Agent based - Useful when you don't have access to the code (i.e. an infrastructure or some SRE's who aren't coders too), or when it's cost/time prohibitive to change each application, and you're willing to have over-instrumentation (not an exhaustive list).
• Instrumentation Libraries - Coded into your application code, getting opinionated Spans/Attributes from the authors of the libraries you use like Database calls and making/receiving HTTP requests.
• Manual - Adding your own context to existing spans and creating new spans around interesting areas of your code that would be useful to understand in production when debugging.

In our docs, we have only "Auto" and "Manual", meaning that engineers feel that to get anything using "Manual" they have to write all the code for all the spans.

I'm not sure of the "right" answer here, I just know that it's an issue, and I'm hearing it more and more (likely a good sign as people are starting to talk about OTel more in my communities).

One thought I had would be use the term "Code based" instead of "Manual". Which I think signifies the idea that you can get what Auto does, but configure it in code rather than via an external application. The other was that we have a standard "Getting Started" page for each language that jumps off into Auto and Code-based detailing reasons to choose each with the specific language idioms in mind (i.e. Java is more of an agent based/AOP style user base so maybe it's more prevalent there to use Auto).

I'd love some thoughts, especially from people outside of the .NET community. If you are putting in opinions though, please let us know which language, and also role like SRE/Application Engineer, you're coming from.

[pellared]

There are usually 3 "things" that is usually important when instrument an application:

1. How you "bootstrap the SDK" - it can be either automatic or manual.
2. How you "instrument libraries" (apply instrumentation libraries) - it can be either automatic or manual
3. How you "instrument your code" - I think this is always manual.

In .NET we found users to use a different mix of these. We even had people bootstrapping the SDK automatically and using plugins to add some manual changes to the default bootstrapping code 😉

[asafm]

Java docs is the same thing:

• Java
  • Getting Started
  • Automatic
  • Manual

I completely agree that as somebody new to OTel, Manual seems like this bad, long option, which I find the opposite.

So instead of:

• Automatic
• Manual

I was thinking of the following options

• External
• Direct

Since the agent is an external way to the JVM to instrument the app and using the library (SDK) it self is the direct way of doing it.

---

• Agent
• Library

Agent is self explanatory - instrumenting via an Java agent, while the second option is instrumenting using the library.

---

My favorite option is:

• Agent
• SDK

Agent is self explanatory - instrumenting via an Java agent, while the second option is instrumenting using the SDK.

Also it's not just about tracing. I find it more severe with metric instrumentation, where you are 100% chance of using the Manual approach since you need your own metric. The word Automatic in java also clash a little bit since in the SDK you have an SDK extension allowing you to Auto Configure based on configuration file / environment variable, so changing Automatic to Agent would also solve that.

[martinjt]

I think the issue with "Agent" is that it's not got the same meaning across all languages. As in, it works for Java, but in .NET, I'm not sure it would make sense to an application engineer.

I think the ship has sailed on "automatic", I don't think anyone is going to allow that to be changed. Manual I don't think has had the same scrutiny so I think that might be the place to make the change.

The key here is that I believe that the OpenTelemetry community want one terminology for all languages/runtimes (I could debate that point, but that isn't the hill I'm going to die on). So it needs to be something that all languages can agree on.

[seh]

I call the non-automatic instrumentation "internal", "integrated", "inline", or "woven".

[martinjt]

"woven"

Now I want to call it "Artisanal Instrumentation"

[maralski]

User instrumentation

[svrnm]

Thanks for raising this issue, @martinjt. Finding the right words for the things we are doing with OpenTelemetry seems to be as old as time, as there have been many many discussions on this topic. I am not saying that it is not worthwhile to have those discussions and to aim for being better, I just want to set the expectations that this is a long long battle :-)

Automatic and Manual as well are defined through the specification and this is the place where it needs to be addressed first:

https://opentelemetry.io/docs/specs/otel/glossary/#manual-instrumentation https://opentelemetry.io/docs/specs/otel/glossary/#automatic-instrumentation (+ https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-library)

One thought I had would be use the term "Code based" instead of "Manual". Which I think signifies the idea that you can get what Auto does, but configure it in code rather than via an external application.

I think one big issue that we have right now is that in people's mind "Automatic Instrumentation" also includes "automatic configuration", "automatic telemetry processing & exporting", "automatic packaging", etc. while in it's truest sense automatic instrumentation only(!) "refers to telemetry collection methods that do not require the end-user to modify application’s source code". We (the docs) are of course part of the problem here, but until we have better language it will remain that way.

I raised a ticket around the same for the spec a while ago but it stalled: https://github.com/open-telemetry/opentelemetry-specification/issues/2866.

Additionally, the lines between "Manual" and "Automatic" are not clear from an end user perspective: I think Node.JS is the best example right now, as you can do a fully automatic instrumentation by loading a pre-built setup of the NodeSDK coming with all the packages, but also a semi-automatic instrumentation by instantiating the NodeSDK and loading it with a bunch of instrumentation libraries or a manual one by using the same NodeSDK without any instrumentation library. To make this clear: this is a good thing for users, as it gives people a single point of entry and they can choose how automatic and how manual things should be. But it is a bad thing for language, since it is really hard to describe it

The other was that we have a standard "Getting Started" page for each language that jumps off into Auto and Code-based detailing reasons to choose each with the specific language idioms in mind (i.e. Java is more of an agent based/AOP style user base so maybe it's more prevalent there to use Auto).

Personal opinion: A "Getting Started" should not make me think, it's here to create appetite: so I don't want to start it with reading why I should do (a) or (b) or when to choose what, what I want from it is a super quick way of getting something tangible with a sample application that I can ideally easily copy into my application to see things happening. That's why there is a bias towards Automatic Instrumentation. People new to OpenTelemetry should see their telemetry (especially traces!) as soon as possible, to have a "Wow!" and be hooked.

But!, it can end with a reasoning about what you should do when and let me know that there are other (maybe better!) options for me. If the getting started was easy and hooked me, I am happy to learn!

We are currently in the progress of standardizing the getting started docs (see https://github.com/open-telemetry/opentelemetry.io/issues/2623), so we can much easier adopt an advanced "What's next" section that gives people insights on automatic vs manual vs hybrid vs etc.

[cartermp]

I've always preferred the term "custom instrumentation"

• Add some custom spans
• Add custom attributes to telemetry

Found myself saying that a lot and people immediately get it.

[martinjt]

Custom works for the spans and attributes.

My point is more about what you call adding the otel packages and configuring it in code versus having IL injection do it in .NET. that's not custom at that stage, as its still getting automatic spans from instrumentation libraries.

[rwkarg]

Software Engineer here (I guess Platform Architect role?) with >15 years in .NET.

Adding code to Startup feels very natural and doesn't leave me guessing as to what's going on. Running some PowerShell/bash commands to "install" and "register" something is an opaque process to me and I don't get any understanding of what's going on.

To me, the Auto instrumentation is a useful tool for scenarios where you don't have access to update a service but want to make a best effort at getting some kind of tracing data out of the service. But that seems to be a niche case, in my experience.

Adding a package, configuring/registering things at startup, and having it all available at design time is a much more natural approach that matches the way I do the rest of my .NET development.

[martinjt]

@svrnm I can understand all those points. My counter is that right now, from an SRE perspective it makes sense. When I say SRE here, I'm talking about those that are more platform than dev, and generally don't change code. That's the focus of our docs, and focused around paradigms that work in the world of say, Java, where an agent is a pretty standard thing.

Ignoring the idioms of each language and runtime is what is turning people off.

In the .NET world, where this is mostly and application engineering effort, Auto-instrumentation from Powershell/bash is just weird. I wouldn't even think about adding the bash script to my base image over adding the ~10 lines of code in my startup. Yet, as an engineer, when I go to the getting started page, I'm presented with Powershell/Bash, and then "manual".

I don't think there's a decision in play on the getting started that's bad. I'm sure that there could be a wording that makes sense, and I'm happy to work on that.

I get the desire for standardising the getting started docs, however that presupposes the audience, that someone will want to go between each of the different languages and see the different approaches, when in fact, the agent is for those people. For the engineers, the language idioms are way more important.

[svrnm]

I get the desire for standardising the getting started docs, however that presupposes the audience, that someone will want to go between each of the different languages and see the different approaches, when in fact, the agent is for those people. For the engineers, the language idioms are way more important.

The assumption is not that someone wants to go between different languages or that they are more ops-vs-devs, the assumption is that a Getting Started should give you something tangible as quick as possible in your preferred language: we have languages (e.g. rust, go & C++ will probably go the same route) using the standardized Getting Started without automation at all. Or JavaScript and Ruby do code-based configuration & "automatic instrumentation" using instrumentation libraries.

If there is a way for .NET to create a getting started that (a) does not make people think and (b) is not weird, then those 2 things are not mutually exclusive. This is something the .NET SIG should look into and decide.

---

To get back to the language discussion: Reading through what has been said here and in other discussions, I want to re-enforce my point that a big issue is that we use "Automatic Instrumentation" (with upper-case letters) in the meaning of "automatic everything", including setup, configuration, instrumentation, exporting, processing, etc. while "automatic instrumentation" is only telemetry collection that does not require the end-user to modify application's source code. So instead of coming up with better words for "manual" vs "automatic" here's a different suggestion:

• In the docs we rename the top level category "Instrumentation" to "Languages". This also solves the problem that newcomers don't know what "Instrumentation" means and are not able to find the docs for their language.
• We have a section called "instrumentation", that describes all the things you do for telemetry collection (add traces, metrics, logs), i.e. mostly what "manual instrumentation" has today
• We have a section (before that) called "Configuration/Setup/..." that describes all the steps you need to do to initialize your SDK (and this can be automatic, load some instrumentation libraries, etc.)
• We rename "automatic" to "zero code" or "no code" or "agent-based" and this is documentation for "Automatic Everything"

I have to think about this myself for a little bit, but I just put it out here for debate.

[james-kaguru]

I agree with what you are saying @svrnm . As a beginner to Otel I had some thoughts on how it could be improved and I felt you have addressed them quite well

In the docs we rename the top level category "Instrumentation" to "Languages". This also solves the problem that newcomers don't know what "Instrumentation" means and are not able to find the docs for their language.

In a way I think we could build on this and ensure that content regarding a specific language is located under its language category. Like for example after "Concepts section" in the navigation ..the next step could be the "Languages" section and move the "Demo" into their respective "Languages".

I believe what you have mentioned the could be great to have as an issue and get additional thoughts from others as well

[svrnm]

I agree with what you are saying @svrnm . As a beginner to Otel I had some thoughts on how it could be improved and I felt you have addressed them quite well

Appreciated, this is a situation where being a newcomer is really helpful, as it is hard to make a good judgment when you know what all those words mean

In the docs we rename the top level category "Instrumentation" to "Languages". This also solves the problem that newcomers don't know what "Instrumentation" means and are not able to find the docs for their language.

In a way I think we could build on this and ensure that content regarding a specific language is located under its language category. Like for example after "Concepts section" in the navigation ..the next step could be the "Languages" section and move the "Demo" into their respective "Languages".

This is probably not feasible since the different components of the demo are highly coupled and should stay together. We are in the progress of building pointers between the language-specific docs and the demo services written in that specific language, so if you feel that there are some places to add those kinds of links, please let us know.

[james-kaguru]

@svrnm I did noticed they are a highly coupled and perhaps they should remain together. However I still believe that the "Languages" section could come first because as a beginner the first thing that was on my mind was how can I generate traces and metrics. The demo is well thought but its a bit advanced for someone who is starting out. I believe the demo serves best as an advanced example so here is my thought process of how id go through the docs as someone whose new.

• I want to log traces and metrics -> I read the getting started page
• The tracing and metrics is working fine is there an example of something that is somewhat production ready? -> I read the examples page.
• The example is working okay. Is there an advanced example that can bring mutliple distributed services together? -> I read the demo section.

[svrnm]

We merged #3761, which solves the "manual" part I raised a separate issue for the "automatic" part, see #3783