search

OpenLiberty / docs

See Open Liberty documentation on https://openliberty.io/docs/
https://openliberty.io/docs/
Other
13 stars 47 forks source link

Documentation of MicroProfile Telemetry #5258

Closed fmhwong closed 1 year ago

fmhwong commented 2 years ago

Epic: https://github.com/OpenLiberty/open-liberty/issues/19172

Content: TBD

dmuelle commented 2 years ago

currently targeting 22.0.0.5 beta/ 22.0.0.6

dmuelle commented 2 years ago

moved to 22.0.0.7

yasmin-aumeeruddy commented 2 years ago

mpTelemetry-1.0 provides support for OpenTelemetry to collect data on the paths that requests take through services.

The use of microservice architecture may increase difficultly to see how services depend on or affect other services. Consequently, making it harder to find the source of latency or inaccuracy.

One way to increase observability of an application is by emitting traces. Traces represent requests and consist of multiple spans. Spans are representative of single operations in a request and contain a name, time-related data, log messages and metadata to give information about what occurs during a transaction.

MicroProfile Telemetry instruments this data in two different ways: Automatic Instrumentation, Manual Instrumentation. The data can then be exported to Jaeger or Zipkin

Automatic Instrumentation Automatic instrumentation allows developers to observe traces without modifying source code in their Jakarta RESTful web service (aka JAX-RS) applications. To start emitting traces in this way, enable mpTelemetry-1.0 in your server.xml. By default, MicroProfile Telemetry tracing is off. To enable any tracing aspects, specify otel.experimental.sdk.enabled=true as a config property or OTEL_EXPERIMENTAL_SDK_ENABLED=true as an environment property for your server.

MpTelemetry-1.0 supports the following system properties/environment variables to further configure the instrumenter:

To export spans to Jaeger, set the following environment variables in server.env.

For more information on these properties, see OpenTelemetry's configuration documentation

Manual Instrumentation

The automatic instrumentation only instrument Jakarta RESTful web service applications. To get further spans on other operations such as database calls, manual instrumentation can be added to those source code.

See OpenTelemetry's manual instrumentation documentation.

Java Agent The OpenTelemetry Java agent enables Java applications to generate and capture telemetry data automatically. To enable the java agent on your Open Liberty application, download the latest version here and add the following to your jvm.options file.

java -javaagent:path/to/opentelemetry-javaagent.jar

The agent is configurable with environment variables and system properties. A list of supported libraries and frameworks can be found in the Java Instrumentation documentation.

fmhwong commented 2 years ago

I added the manual instrumentation section and some more details in other sections.

dmuelle commented 2 years ago

Hi @fmhwong @yasmin-aumeeruddy , thanks for providing this info. I have a couple questions to get started.

Thanks!

yasmin-aumeeruddy commented 2 years ago

Hi @dmuelle

OpenTelemetry must be configured by MicroProfile Config so variables set in the server.xml are accepted.

otel.experimental.sdk.enabled

otel.service.name

otel.traces.exporter

otel.exporter.jaeger.endpoint

otel.exporter.otlp.endpoint

otel.exporter.zipkin.endpoint

otel.exporter.jaeger.timeout

There is a spec doc from Eclipse here.

dmuelle commented 2 years ago

Thanks @yasmin-aumeeruddy. I have a couple questions-

To export spans to Jaeger, set the following environment variables in server.env.

Do you have to use these environment variables to export Jaeger? Are they required in addition to setting the corresponding config properties or is it either/or?

Thanks

dmuelle commented 2 years ago

Hi @yasmin-aumeeruddy @fmhwong , the initial draft of the MicroProfile Telemetry doc is ready for review:

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html

The MicroProfile Config properties for MP OT are documented here:

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-config-properties.html#telemetry

Please let me know what changes are needed. One thing I want to check is the values in the property examples. Several examples specify the default values for properties, which might be confusing as I don't think you'd need to set a property to use the default value in most cases.

When you're satisfied with the doc, you can add the Technical Reviewed label to this issue to sign off and the doc will publish with whatever release MP OT is GA (currently projected 22.0.0.13). Thanks

yasmin-aumeeruddy commented 1 year ago

Hi @dmuelle I hadn't mentioned OpenTracing's deprecation:

MicroProfile OpenTracing features will be stabilized.​

mpOpenTracing-1.x, openTracing-1.x​

mpOpenTracing-2.0, openTracing-2.0​

mpOpenTracing-3.0, openTracing-3.0

dmuelle commented 1 year ago

@yasmin-aumeeruddy - thanks for bringing this up- do you know whether we need POC approval for those stabilizations? And if so, will they be stabilized by the time of MP Telemetry Ga?

For the docs, we wont have to do much other than add a stabilization statement to the feature pages, but that's pending POC approval, if needed.

We do have a couple of Guides that use OpenTracing but I'm not sure how the guides team handles deprecations. We can check with @gkwan-ibm for more info

gkwan-ibm commented 1 year ago

I need to discuss with @yeekangc. Either

dmuelle commented 1 year ago

see also w/r/t openTracing stabilization:

issue in main OL repo: https://github.com/OpenLiberty/open-liberty/issues/23422

we will add stabilization statements to the OpenTracing and MicroProfile OpenTracing pages- unless there are updates planned for the metatype that will add these statements as part of the autogenerated feature descriptions. If the stabilization is the same release as MP OpenTelemetry, we can track those updates in this issue. Otherwise, I'll open a new issue.

dmuelle commented 1 year ago

opened https://github.com/OpenLiberty/docs/issues/6104 to track the openTracing stabilization

fmhwong commented 1 year ago

A few changes are needed:

Change "To enable tracing, specify either the otel.experimental.sdk.enabled=true MicroProfile Config property or the OTEL_EXPERIMENTAL_SDK_ENABLED=true environment variable for your server." To "To enable tracing, specify either the otel.sdk.disabled=false MicroProfile Config property or the OTEL_SDK_DISABLED=false environment variable for your server."

In general, change all occurrences of "otel.experimental.sdk.enabled=true" to "otel.sdk.disabled=false".

The link is broken below: "To start emitting traces with automatic instrumentation, enable the MicroProfile Telemetry feature in your server.xml"

dmuelle commented 1 year ago

Thanks @fmhwong - I've updated to otel.sdk.disabled=false in both the MP Telemetry topic and the MP config properties reference page (assuming the default here is true).

The link to the MP Telemetry feature will be active after GM when we have the autogenerated feature page available- I will verify the link at that time.

Azquelt commented 1 year ago

Hi @dmuelle:

While testing MP Telemetry with the OpenTelemetry agent, we found some limitations which should be documented.

I also think we need to be a little clearer in the documentation that the Java agent is a tool provided by the opentelemetry project. It's not part of liberty and we don't support it but can be used with liberty. We should also be a bit clearer about what it actually does. I'll have a go at writing something for this tomorrow.

Reading through the draft page, I've also noticed that we haven't given the user enough information on how to do manual instrumentation. You need to enable third-party APIs and use CDI to inject the Tracer object. I'll follow up with more information on this section shortly.

Azquelt commented 1 year ago

@dmuelle : Here's some suggested text for the Agent instrumentation section, further doc for the manual instrumentation section still to come.

Agent instrumentation

The OpenTelemetry Instrumentation for Java project provides a Java agent JAR that can be attached to any Java 8+ application and dynamically injects bytecode to add telemetry support to popular open source libraries and frameworks. If you are using any of the supported libraries in your application, you can use this agent with OpenLiberty to instrument them.

To do so, download the latest agent version and add the following line to your jvm.options file:

-javaagent: path/to/opentelemetry-javaagent.jar

Note: ^^ this corrects a mistake in the current draft page

However, you need to be aware of the following limitations when using the agent:

Note: The Java 2 Security link goes to IBM KC for Websphere Liberty because I couldn't find a documentation page for it in the Open Liberty docs. It really means using the Java Security Manager, I don't think it's widely used and Security Manager is deprecated and being removed in newer versions of Java, but we need to have a link here so that users know what it is, otherwise not being compatible with security sounds scary.

Azquelt commented 1 year ago

@dmuelle: Here's some suggested text for the Manual instrumentation section.

Manual Instrumentation

Automatic instrumentation is available only for Jakarta RESTful web service applications. To create spans for other operations, such as database calls, you can add manual instrumentation to the source code for those operations using the OpenTelemetry API. However, before manually instrumenting your code, you must first follow these steps:

  1. Enable third-party APIs for your application in server.xml

    <webApplication id="app-name" location="app-name.war">
    <classloader apiTypeVisibility="+third-party"/>
</webApplication>

  2. Add the opentelemetry API as a provided dependency to your build path. For example with maven:

    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.19.0</version>
    <scope>provided</scope>
</dependency>

Now you're ready to instrument your code. Here are some examples using the Open Telemetry API:

Add additional information (e.g. the user ID) to the current span. Any information you add to a span will be visible when you look at traces on your trace server.

private static final AttributeKey<String> USER_ID_ATTR = AttributeKey.stringKey("userId");

@Inject private Span currentSpan;

@GET
public String myMethod() {
    ...
    currentSpan.setAttribute(USER_ID_ATTR, getUserId());
    ...
}

Create a subspan around a particular operation (e.g. querying a database). This will allow you to see how long it took and the order in which it occurred relative to other spans. 

@Inject private Tracer tracer;

@GET
public String myMethod() {
    ...
    Span newSpan = tracer.spanBuilder("QueryDatabase").startSpan();
    try (Scope s = newSpan.makeCurrent()) {
        queryDatabase();
    } finally {
        newSpan.end();
    }
    ...
}

IMPORTANT: You must call .end() on any span you create, otherwise the span won't be recorded.

IMPORTANT: The "current" span is used as the parent for any new spans that are created so usually when you create a span you also want to make it current. However, you must close the Scope returned by Span.makeCurrent() which is mostly easily done with a try-with-resources block as in the example above.

More examples of adding manual instrumentation can be found in the OpenTelemetry manual instrumentation documentation. However, when reading this documentation, bear in mind that when using the mpTelemetry feature in liberty, you must obtain the OpenTelemetry and Tracer objects by injecting them, not by creating your own.

I'm not sure whether you use the admonitions (IMPORTANT, CAUTION etc.) in the OL doc. If not, you'll have to find the best way to draw attention to those important points.

dmuelle commented 1 year ago

Hi @Azquelt - the updated draft is ready for review when you have a chance:

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html

Let me know if any further edits are needed. If not, you can sign off by adding the Technical reviewed label to this issue. Thanks!

yasmin-aumeeruddy commented 1 year ago

@dmuelle The manual instrumentation section needs some info about the @WithSpan annotation

For example:

@ApplicationScoped
class SpanBean {

    @WithSpan("name")
    void spanName() {
       ...
    }

    @WithSpan
    void spanArgs(@SpanAttribute(value = "arg") String arg) {
       ...
    }
}

Then at the end of the manual instrumentation section:

For more information, see the OpenTelemetry manual instrumentation documentation. However, remember that when you use the MicroProfile Telemetry feature in Open Liberty, you must obtain the OpenTelemetry and Tracer objects by injecting them, not by creating your own. Be aware that the documentation provides information for the OpenTelemetry Metrics and Logging APIs which are not supported by MicroProfile Telemetry.

dmuelle commented 1 year ago

Thanks @yasmin-aumeeruddy - draft is updated with this info:

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html#manual

Azquelt commented 1 year ago

@dmuelle We noticed a few issues with the Automatic instrumentation section:

Taking account of those points, here's my suggested update for this section:

With automatic instrumentation, you can observe traces without modifying the source code in your Jakarta RESTful web service (formerly JAX-RS) applications. To start emitting traces with automatic instrumentation, enable the MicroProfile Telemetry feature in your server.xml file. By default, MicroProfile Telemetry tracing is off. To enable tracing, specify the otel.sdk.disabled=false MicroProfile Config property as well as any exporter configuration needed for your tracing service service.

For example, to export traces to a Jaeger server with OTLP enabled, you could add these entries to bootstrap.properties:

otel.sdk.disabled=false
otel.traces.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4317/

To export traces to a Zipkin server, you could use these properties instead:

otel.sdk.disabled=false
otel.traces.exporter=zipkin
otel.exporter.zipkin.endpoint=http://localhost:9411/api/v2/spans

You can configure how MicroProfile Telemetry collects and exports traces by specifying configuration properties in any of the config sources that are available to MicroProfile Config. If using environment variables, you should uppercase the key name and convert any punctuation to underscores. For example otel.sdk.disabled=false would become OTEL_SDK_DISABLED=false.

For more information about the available properties, see MicroProfile Config properties: MicroProfile Telemetry.

Note: I've deliberately dropped the final paragraph as it's not helpful. We support one list of properties and they may be provided from any MP Config source. Linking to a different spec for environment variables doesn't make sense.

I'm a little unsure of the content of that last link: https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-config-properties.html#telemetry

The properties are in an unhelpful order and there are loads that aren't listed. We actually support most of the properties listed here that apply to tracing: https://github.com/open-telemetry/opentelemetry-java/tree/v1.19.0/sdk-extensions/autoconfigure

Additionally on that page we link to the spec and the link should be updated to the final spec and the correct section.

I guess we should update our page with a better list of supported properties. I'll try to do that tomorrow morning.

dmuelle commented 1 year ago

Thanks @Azquelt , I've updated the Automatic instrumentation section per your comments. I also reordered the MP config properties to be alphabetical (which they shouldve been already). If theres a better order for them just let me know. I updated the spec link on that page to point to 1.0

yasmin-aumeeruddy commented 1 year ago

Sorry @dmuelle I have noticed another problem with the manual instrumentation. Step 2 needs another dependency: 

    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.19.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-instrumentation-annotions</artifactId>
        <version>1.19.0-alpha</version>
        <scope>provided</scope>
    </dependency>
Azquelt commented 1 year ago

Ah that's true, the @WithSpan example needs opentelemetry-instrumentation-annotions

Azquelt commented 1 year ago

For the MP Config properties, I think we either need to include all the properties we support, or we should be clearer that the list only includes the most important properties and link off for the full list.

If the properties list is meant to be alphabetical then that is ok. If we were to expand the list to include all properties, I would split it into logical sections as it is in the Autoconfigure SDK documentation because the list is quite long.

Assuming we keep the shorter list we have at the moment, I would like to make the following changes:

Azquelt commented 1 year ago

Reading through the latest draft of the Distributed Tracing page, I think the content is all good, apart from needing Yasmin's change above.

There's one formatting error: the last example code block doesn't align with the others. I suspect it's just missing a + to make it part of the bullet point above it.

dmuelle commented 1 year ago

Thanks @yasmin-aumeeruddy @Azquelt - I've added these changes to the drafts

Azquelt commented 1 year ago

Thanks @dmuelle

I just noticed that the example for otel.exporter.otlp.endpoint is otel.exporter.jaeger.endpoint=http://localhost:4317

It should be otel.exporter.otlp.endpoint=http://localhost:4317

dmuelle commented 1 year ago

Thanks for catching that- fixed in #6322

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-config-properties.html#telemetry

Azquelt commented 1 year ago

You can configure how MicroProfile Telemetry collects and exports traces by specifying configuration properties in any of the config sources that are available to MicroProfile Config. Alternatively, to use environment variables instead of properties, make the key name uppercase and convert any punctuation to underscores. For example, the otel.sdk.disabled=false property is equivalent to the OTEL_SDK_DISABLED=false environment variable.

I find the wording "Alternatively, to use environment variables instead of properties..." misleading here.

Environment variables are one of the default config sources for MicroProfile Config, environment variables are used as config properties, they're not an alternative.

However, since there are restrictions on the characters allowed in an environment variable name, MP Config allows a looser match for config property names. When looking up the value for the property otel.sdk.disabled, it will consider the value of an environment variable named OTEL_SDK_DISABLED.

I would change the wording here to "If using environment variables, " or "If setting these properties via environment variables, ".

This sentence is really just a hint to newer users, users who are familiar with MP Config should know this already, but we don't want to either mislead new users or confuse existing users about how environment variables relate to MP Config properties.

dmuelle commented 1 year ago

Hi @Azquelt @yasmin-aumeeruddy

I updated the draft per Andrew's last comment and also added a link to the "Differences..." topic

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html

dmuelle commented 1 year ago

Hi @Azquelt @yasmin-aumeeruddy- are any further changes needed for this issue? If not, you can add the Technical reviewed label to this issue and the topic will publish with 23.0.0.2 GA (unless the feature is moved to a later release).

Azquelt commented 1 year ago

Hi @dmuelle,

I've reviewed and spotted one typo on the MP Telemetry docs page

You can enable MicroPRofile Telemetry

As this is not a technical issue, I've added the Technical reviewed label.

dmuelle commented 1 year ago

Thanks @Azquelt - fixed that typo in #6442

dmuelle commented 1 year ago

@ramkumar-k-9286 this is ready for peer review

ramkumar-k-9286 commented 1 year ago

Peer Review

Link to MicroProfile Telemetry should be done in the first instance of it being mentioned. Right? MicroProfile Telemetry is mentioned in the first paragraph.

MicroProfile Telemetry is based on the OpenTelemetry project, which is a collection of open source vendor-agnostic tools, APIs, and SDKs for creating and managing trace data. -> MicroProfile Telemetry is based on the OpenTelemetry project, which is a collection of open-source vendor-agnostic tools, APIs, and SDKs for creating and managing trace data.

For information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0. -> For more information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see the Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0.

To enable tracing, specify the otel.sdk.disabled=false MicroProfile Config property and any exporter configuration that your tracing service service requires. -> To enable tracing, specify the `otel.sdk.disabled=false MicroProfile Config property and any exporter configuration that your tracing service requires.

For example, to export traces to a Jaeger server with the OpenTelemetry Protocol (OTLP) enabled, add the following entries to your bootstrap.properties file: -> For example, to export traces to a Jaeger server with the OpenTelemetry Protocol (OTLP) enabled, add the following entries to your bootstrap.properties file.

To export traces to a Zipkin server, you can use the following properties instead: -> To export traces to a Zipkin server, you can use the following properties instead .

Enable third-party APIs for your application by adding the following code in your server.xml file: -> Enable third-party APIs for your application by adding the following code in your server.xml file .

For example, with Maven, add the following code to your pom.xml file: -> For example, add the following code to your pom.xml file with Maven .

The following examples show configuration with the OpenTelemetry API. -> The following examples show the configuration with the OpenTelemetry API.

Any information you add to a span is visible when you look at traces on your trace server. -> Any information that you add to a span is visible when you look at traces on your trace server.

The following important considerations apply to manual instrumentation: -> The following important considerations apply to manual instrumentation .

You must call the .end() method on any span you create, otherwise the span won’t be recorded. ( Acrolinx suggestion) -> You must call the .end() method on any span you create, otherwise the span is not recorded.

The OpenTelemetry Instrumentation for Java project provides a Java agent JAR file that can be attached to any Java 8+ application to dynamically inject bytecode that adds telemetry support to popular open source libraries and frameworks. ( Acrolinx - too long) -> The Java agent JAR file offered by the OpenTelemetry Instrumentation for Java project can be added to Java 8+ applications. This dynamically injects bytecode to enable telemetry support for well-known open-source libraries and frameworks.

The OpenTelemetry Java agent is a tool that is provided by the OpenTelemetry project. Although it is compatible with Open Liberty, it is a separate project and is subject to the following limitations: Configuration works differently when you use the agent. Configuration of the agent is well documented, but the following aspects are different from configuration without the agent: -> The OpenTelemetry Java agent is a tool that is provided by the OpenTelemetry project. Although it is compatible with Open Liberty, it is a separate project and is subject to the following limitations . The configuration works differently when you use the agent. Configuration of the agent is well documented, but the following aspects are different from configuration without the agent .

Because the agent reads its configuration very early in the startup process, -> Because the agent reads its configuration early in the startup process,

Open Liberty uses many open source libraries internally. -> Open Liberty uses many open-source libraries internally.

dmuelle commented 1 year ago

Thanks for reviewing @ramkumar-k-9286, all suggestions implemented except: