Logback appender needs to be re-initialized after spring application starts

[lmolkova]

Describe the bug

I have the following setup

• manual instrumentation with opentelemetry-logback-appender-1.0
• logback appender is initialized statically before SpringApplication.run is called
• no logs are exported

Steps to reproduce

@SpringBootApplication
public class DemoApplication implements ApplicationRunner {
    private final static Logger LOGGER = LoggerFactory.getLogger(DemoApplication.class);
    private final static OpenTelemetry OTEL = initOTel();

    private static OpenTelemetry initOTel() {
        System.setProperty("otel.traces.exporter", "none");
        System.setProperty("otel.metrics.exporter", "none");
        System.setProperty("otel.logs.exporter", "logging");

        OpenTelemetry otel = AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build()
                .getOpenTelemetrySdk();
        OpenTelemetryAppender.install(otel);

        return otel;
    }

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Override
    public void run(ApplicationArguments args) {
        // uncomment next line to make things work
        // OpenTelemetryAppender.install(OTEL);
        LOGGER.info("Starting demo application");
    }
}

full code: demo.zip

Expected behavior

if I uncomment OpenTelemetryAppender.install(OTEL), things work as expected:

2024-01-22T21:47:20.22Z INFO 'Starting demo application' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: com.example.demo.DemoApplication:] {}

Actual behavior

nothing is logged

Javaagent or library instrumentation version

1.34.1

Environment

JDK: 21 OS: windows

Additional context

Spring re-creates log appenders during SpringApplication.run call

As a result, this new OTelAppender never gets configured with otel instance

I believe it's possible to apply global otel in the constructor or default to it until another instance is provided.

Otherwise, this behavior should at least be documented (ideally as a spring-boot example with proper way to config otel there).

[laurit]

@jeanbisutti could you take a look

[jeanbisutti]

As you mentioned, Spring Boot re-initializes the logging system. So, the way to proceed with a programmatic use of the SDK is to install OpenTelemetry in the OpenTelemetry appender after this re-initialization.

However, with Spring Boot, the OpenTelemetry documentation recommends using the OpenTelemetry Java agent or the OpenTelemetry starter, not a programmatic use of the SDK.

The OpenTelemetry starter automatically injects an OpenTelemetry instance into the OpenTelemetry Logback appender. A PR was created to remove the need to add the OpenTelemetry appender in the Logback xml file. So, with the OpenTelemetry starter having this PR code, the logging instrumentation will be available without configuration for Logback.

The opentelemetry-java-examples/spring-native shows an example with the OpenTelemetry starter.

[lmolkova]

nice! thank you @jeanbisutti for the detailed response and sharing the example.