BatchLogRecordProcessorBuilder doesn't force flush logs to export when the maxQueueSize is full.

[tongshushan]

Hello,

I set the 2 parameters of BatchLogRecordProcessorBuilder as below: setScheduleDelay(Duration.ofMinutes(2)).setMaxQueueSize(10) , and I sent more than 100 logs in 1 minutes, but after 2 minutes the exporter only receive 8 logs, many logs lost. But if I don't modify the maxQueueSize, left it as default, the exporter can receive all logs.

Configuration Code:

@Bean
    public OpenTelemetry openTelemetry() {
        Resource resource = Resource.getDefault()
                .merge(Resource.create(Attributes.builder()
                        .put("test", "xxx")
                        .put(ResourceAttributes.SERVICE_NAME, serviceName)
                        .build()));
        System.setProperty("otel.service.name", serviceName);
        System.setProperty("otel.exporter.otlp.endpoint", oltpLogEndpoint);
        System.setProperty("otel.logs.exporter", "otlp");
        OpenTelemetrySdk openTelemetrySdk = AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();
        SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
                .addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().setEndpoint(oltpLogEndpoint).build()).build())
                .setResource(resource)
                .build();
        SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
                .registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().setEndpoint(oltpLogEndpoint).build()).build())
                .setResource(resource)
                .build();
        SdkLoggerProvider sdkLoggerProvider = SdkLoggerProvider.builder()
                .setResource(resource)
                .addLogRecordProcessor(BatchLogRecordProcessor.builder(OtlpGrpcLogRecordExporter.builder().setEndpoint(oltpLogEndpoint).build()).setScheduleDelay(Duration.ofMinutes(2)).setMaxQueueSize(10).build())
                .build();
        OpenTelemetrySdk upgradeOpenTelemetrySdk = openTelemetrySdk.builder()
                .setMeterProvider(sdkMeterProvider)
                .setTracerProvider(sdkTracerProvider)
                .setLoggerProvider(sdkLoggerProvider)
                .build();
        // Install OpenTelemetry in logback appender
        io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender.install(upgradeOpenTelemetrySdk);

        // Route JUL logs to slf4j
        SLF4JBridgeHandler.removeHandlersForRootLogger();
        SLF4JBridgeHandler.install();
        return upgradeOpenTelemetrySdk;
    }

Caller code:

    @Value("${spring.application.name}")
    private String serviceName;

   @Autowired
    OpenTelemetry openTelemetry;

    LoggerProvider loggerProvider;

    @Autowired
    ApiAopHelper(OpenTelemetry openTelemetry) {
        loggerProvider = openTelemetry.getLogsBridge();
    }

//just ignore the bussiness fields here
 public void log2Otlp(args ...) {
     Logger logger = loggerProvider.get(serviceName);
        DataloaderLog dataloaderLog = new DataloaderLog();
        dataloaderLog.setCallApiMethodName(caller);
         ...
        AttributeKey<Long> durationKey = AttributeKey.longKey("duration");
        ...
        AttributeKey<String> response = AttributeKey.stringKey("response");

        Gson gson = new Gson();
        dataloaderLog.setDuration(duration);
        dataloaderLog.setResponse(status);
        LogRecordBuilder logRecordBuilder = logger.logRecordBuilder()
                .setSeverity(Severity.INFO)
                .setTimestamp(Instant.now())
               ...
                .setAttribute(durationKey, duration)
                .setAttribute(response, status);
        String body = "callApiMethodName {callApiMethodName} .... duration {duration} ms and response {response}";
        log.info(gson.toJson(dataloaderLog));
        logRecordBuilder.setBody(body);
        logRecordBuilder.emit();
   }

Steps to reproduce Call method log2Otlp(args ...) more than 100 times to send logs.

Environment java 8 io.opentelemetry: 1.37.0

[tongshushan]

I find the root cause is maxQueueSize < maxExportBatchSize. Is it works as designed? If it is ,can the api give some clear message if misconfigued?

[jkwatson]

This is working as designed. I think logging a warning when this mis-configuration is done is a reasonable request. Can you put in a PR to add a log message, @tongshushan ?

[tongshushan]

Yes, I put the PR here: https://github.com/open-telemetry/opentelemetry-java/issues/6454

Thanks.