Support OpenTelemetry Log signal

[brunobat]

Description

We need to support the OpenTelemetry logging signal available at: https://opentelemetry.io/docs/instrumentation/java/ The spec/impl is still under development (at the creation of this issue it's still experimental) and work should after it's marked as stable.

Depends on #29911

Implementation ideas

No response

[quarkus-bot[bot]]

/cc @radcortez (opentelemetry)

[t1]

According to the linked page, the spec is now stable 😁

[brunobat]

Yes @t1. But supporting it on Quarkus will take some time to implement and it's not in the shot term roadmap.

[t1]

I'm probably not deep enough into OTel, but isn't this about transporting log statements? If it is, I think this is crucial. I'm currently looking for a future-proof solution, and OTel looks very promising. But if Quarkus supports only traces and neither metrics nor logs (not to speak of profiling), I'll have to look for alternatives. Or do I get it wrong?

[brunobat]

OTel OTLP protocol output will be standard.

There's native OTel Tracing support.

You can already get Micrometer metrics output using the OTel's OTLP protocol and receive that in the standard OTel collector. Micrometer just provides a more convenient and complete API to define metrics.

In relation to logs, there will be the need to forward current loggers output to OTel. That work will be done later. OTel Metrics instrumentation support will be added after the Observation API is implemented.

[luciantimar]

Hi @brunobat,

To get logs exported to an OTEL collector is it possible to use some of the logs appenders provided in the examples OpenTelemetry Log Appenders ?

I know that Quarkus relies on the JBoss Logging library but is it possible to use the same approach?

[brunobat]

That's the plan, however, logging is not yet supported by the quarkus-opentelemetry extension. It's the target of this issue.

[crumohr]

@brunobat is there any update regarding the roadmap to implement this feature or any way to support?

[luciantimar]

Hi,

is not the best approach, but until the feature is ready we have managed to have the logs also exported by overwriting the default Open telemetry bean and providing our own addLoggerProviderCustomizer

import io.opentelemetry.api.OpenTelemetry;
import io.quarkus.arc.properties.IfBuildProperty;
import io.quarkus.runtime.StartupEvent;
import io.smallrye.config.Priorities;
import jakarta.annotation.Priority;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
import jakarta.inject.Singleton;

@Singleton
@IfBuildProperty(name = "quarkus.otel.enabled", stringValue = "true")
public class OpenTelemetryStartup {
  @Inject OpenTelemetry openTelemetry;

  void onStart(@Priority(Priorities.APPLICATION - 1) @Observes StartupEvent event) {
    // need @Inject to override OpenTelemetry default bean
  }
}

import static java.lang.Boolean.TRUE;
import static java.util.Collections.emptyList;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.IdGenerator;
import io.opentelemetry.sdk.trace.SpanProcessor;
import io.opentelemetry.sdk.trace.samplers.Sampler;
import io.quarkus.arc.properties.IfBuildProperty;
import io.quarkus.opentelemetry.runtime.config.build.OTelBuildConfig;
import io.quarkus.opentelemetry.runtime.config.runtime.OTelRuntimeConfig;
import io.quarkus.opentelemetry.runtime.tracing.DelayedAttributes;
import io.quarkus.opentelemetry.runtime.tracing.DropTargetsSampler;
import io.quarkus.opentelemetry.runtime.tracing.TracerRecorder;
import io.quarkus.opentelemetry.runtime.tracing.TracerUtil;
import io.quarkus.runtime.ApplicationConfig;
import io.smallrye.config.ConfigValue;
import io.smallrye.config.NameIterator;
import io.smallrye.config.SmallRyeConfig;
import jakarta.enterprise.inject.Any;
import jakarta.enterprise.inject.Instance;
import jakarta.enterprise.inject.Produces;
import jakarta.inject.Inject;
import jakarta.inject.Singleton;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.eclipse.microprofile.config.ConfigProvider;

// TODO: copied from
// https://github.com/quarkusio/quarkus/blob/3.2.7.Final/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/OpenTelemetryProducer.java
//  will be removed when quarkus adds support for logging, also remove from, also
// remove OpenTelemetryStartup
@Singleton
public class OpenTelemetryProducer {
  @Inject Instance<IdGenerator> idGenerator;
  @Inject @Any Instance<Resource> resources;
  @Inject @Any Instance<DelayedAttributes> delayedAttributes;
  @Inject @Any Instance<Sampler> sampler;
  @Inject @Any Instance<SpanProcessor> spanProcessors;
  @Inject OTelBuildConfig oTelBuildConfig;
  @Inject OTelRuntimeConfig oTelRuntimeConfig;
  @Inject ApplicationConfig appConfig;

  @Produces
  @IfBuildProperty(name = "quarkus.otel.enabled", stringValue = "true")
  @Singleton
  public OpenTelemetry getOpenTelemetry() {
    final Map<String, String> oTelConfigs = getOtelConfigs();

    if (oTelRuntimeConfig.sdkDisabled()) {
      return AutoConfiguredOpenTelemetrySdk.builder()
          .setResultAsGlobal(true)
          .registerShutdownHook(false)
          .addPropertiesSupplier(() -> oTelConfigs)
          .build()
          .getOpenTelemetrySdk();
    }

    final AutoConfiguredOpenTelemetrySdk autoConfiguredOpenTelemetrySdk =
        AutoConfiguredOpenTelemetrySdk.builder()
            .setResultAsGlobal(true)
            .registerShutdownHook(false)
            .addPropertiesSupplier(() -> oTelConfigs)
            .setServiceClassLoader(Thread.currentThread().getContextClassLoader())
            // no customization needed for spanExporter. Loads SPI from CDI
            .addResourceCustomizer(
                (existingResource, configProperties) -> {
                  if (oTelBuildConfig.traces().enabled().orElse(TRUE)) {
                    Resource consolidatedResource =
                        existingResource.merge(
                            Resource.create(delayedAttributes.get())); // from cdi

                    // if user explicitly set 'otel.service.name', make sure we don't override it
                    // with defaults
                    // inside resource customizer
                    String serviceName =
                        oTelRuntimeConfig
                            .serviceName()
                            .filter(sn -> !sn.equals(appConfig.name.orElse("unset")))
                            .orElse(null);

                    // Merge resource instances with env attributes
                    Resource resource =
                        resources.stream()
                            .reduce(Resource.empty(), Resource::merge)
                            .merge(
                                TracerUtil.mapResourceAttributes(
                                    oTelRuntimeConfig.resourceAttributes().orElse(emptyList()),
                                    serviceName)); // from properties
                    return consolidatedResource.merge(resource);
                  } else {
                    return Resource.builder().build();
                  }
                })
            .addSamplerCustomizer(
                (existingSampler, configProperties) -> {
                  if (oTelBuildConfig.traces().enabled().orElse(TRUE)) {
                    final Sampler effectiveSampler =
                        sampler.stream()
                            .findFirst()
                            .map(Sampler.class::cast) // use CDI if it exists
                            .orElse(existingSampler);

                    // collect default filtering targets (Needed for all samplers)
                    List<String> dropTargets = new ArrayList<>();
                    if (oTelRuntimeConfig
                        .traces()
                        .suppressNonApplicationUris()) { // default is true
                      dropTargets.addAll(TracerRecorder.dropNonApplicationUriTargets);
                    }
                    if (!oTelRuntimeConfig.traces().includeStaticResources()) { // default is false
                      dropTargets.addAll(TracerRecorder.dropStaticResourceTargets);
                    }

                    // make sure dropped targets are not sampled
                    if (!dropTargets.isEmpty()) {
                      return new DropTargetsSampler(effectiveSampler, dropTargets);
                    } else {
                      return effectiveSampler;
                    }
                  } else {
                    return Sampler.alwaysOff();
                  }
                })
            .addTracerProviderCustomizer(
                (builder, configProperties) -> {
                  if (oTelBuildConfig.traces().enabled().orElse(TRUE)) {
                    idGenerator.stream().findFirst().ifPresent(builder::setIdGenerator); // from cdi
                    spanProcessors.stream().forEach(builder::addSpanProcessor);
                  }
                  return builder;
                })
            .addLoggerProviderCustomizer(
                (existing, configProperties) ->
                    existing.addLogRecordProcessor(
                        BatchLogRecordProcessor.builder(OtlpGrpcLogRecordExporter.getDefault())
                            .build()))
            .build();
    return autoConfiguredOpenTelemetrySdk.getOpenTelemetrySdk();
  }

  private Map<String, String> getOtelConfigs() {
    Map<String, String> oTelConfigs = new HashMap<>();
    SmallRyeConfig config = ConfigProvider.getConfig().unwrap(SmallRyeConfig.class);

    // instruct OTel that we are using the AutoConfiguredOpenTelemetrySdk
    oTelConfigs.put("otel.java.global-autoconfigure.enabled", "true");

    // load new properties
    for (String propertyName : config.getPropertyNames()) {
      if (propertyName.startsWith("quarkus.otel.")) {
        ConfigValue configValue = config.getConfigValue(propertyName);
        if (configValue.getValue() != null) {
          NameIterator name = new NameIterator(propertyName);
          name.next();
          oTelConfigs.put(name.getName().substring(name.getPosition() + 1), configValue.getValue());
        }
      }
    }
    return oTelConfigs;
  }
}

Best regards Lucian

[brunobat]

@loicmathieu can you submit your draft PR for this issue?

[loicmathieu]

Hi, I'll try to submit my PR tomorrow so we can move forward on this topic.

[loicmathieu]

Hi, draft PR is here: https://github.com/quarkusio/quarkus/pull/38239

[blazmrak]

Quarkus >3.3.0 changed how OpenTelemetry is produced. How can I export the logs to OpenTelemetry collector? Should I use the agent that just exports the logs and disable trace/metrics instrumentation?

[alexalmeida52]

I need to display logs in Jaeger, but I'm not able to export them with opentelemetry. Do you already have a solution for this?

[loicmathieu]

@alexalmeida52 you can follow https://github.com/quarkusio/quarkus/pull/38239 that shoud provide this functionality.

For the moment, there is some work to be done at the Quarkus OpenTelemetry implementation to be able to send logs via the logs bridge so the PR is blocked until this work is done.

[brunobat]

@alexalmeida52, as far as I know, Jaeger is not able to display logs, just traces. But you should check that in https://github.com/jaegertracing/jaeger