Very slow loki responses when using JDK11 version - compared to JDK8

[czterocyty]

Hi,

I used before appender for JDK8 with ApacheHttpClient:

Gradle

    implementation "com.github.loki4j:loki-logback-appender-jdk8:1.2.0"
    implementation "org.apache.httpcomponents:httpclient:4.5.13"

Part of logback.xml

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http class="com.github.loki4j.logback.ApacheHttpSender">
            <url>${LOKI_URL}</url>
        </http>
        <format>
            <label>
                <pattern>job=foo,container=${HOSTNAME},env=${LOKI_APP_ENVIRONMENT}</pattern>
            </label>
            <message>
                <pattern>l=%level c=%logger{36} t=%thread | %msg %ex</pattern>
            </message>
            <sortByTime>true</sortByTime>
            <staticLabels>true</staticLabels>
        </format>

        <metricsEnabled>true</metricsEnabled>
        <verbose>true</verbose>
    </appender>

but then switched to version JDK 11.0 and experienced some problems with send actions. The pipeline was very slow, metrics over loki4j_send_time_seconds_max showed me 120 seconds. But even from logs batch DRAIN and batch SEND event were 4 minutes apart.

I was experiencing with direct buffers, batch timeout batch size but results were the same.

    implementation "com.github.loki4j:loki-logback-appender:1.2.0"

OS Version: Ubuntu 20.04 Loki server: against 1.6.1 and 2.2.1 self-hosted Frameworks used: Micronaut 2.5.5 with Micrometer 1.6.6 and Micromter-prometheus

Some glue point to make metrics of loki4j exposes through Prometheus endpoint:

import io.micrometer.core.instrument.Metrics
import io.micrometer.core.instrument.composite.CompositeMeterRegistry
import io.micronaut.configuration.metrics.aggregator.MeterRegistryConfigurer
import javax.inject.Singleton

@Singleton
class Loki4jMeterRegistryConfigurer : MeterRegistryConfigurer<CompositeMeterRegistry> {

    override fun configure(meterRegistry: CompositeMeterRegistry) {
        Metrics.globalRegistry.add(meterRegistry)
    }

    override fun getType(): Class<CompositeMeterRegistry> {
        return CompositeMeterRegistry::class.java
    }
}

Logback debug log: loki4j.txt.gz

[nehaev]

Hi @czterocyty,

Thanks for reporting! Could you please switch to JavaHttpSender by modifying your config the following way:

<http class="com.github.loki4j.logback.JavaHttpSender">

and check if the problem is still reproducible?

[czterocyty]

Hi @nehaev

I had the following configuration of logback for JDK11 client:

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>${LOKI_URL}</url>
        </http>
        <format>
            <label>
                <pattern>job=foo,container=${HOSTNAME},env=${LOKI_APP_ENVIRONMENT}</pattern>
            </label>
            <message>
                <pattern>l=%level c=%logger{36} t=%thread | %msg %ex</pattern>
            </message>
            <sortByTime>true</sortByTime>
            <staticLabels>true</staticLabels>
        </format>

        <metricsEnabled>true</metricsEnabled>
        <verbose>true</verbose>
    </appender>

I assume that missing class under http tag means com.github.loki4j.logback.JavaHttpSender by default.

[nehaev]

I assume that missing class under http tag means com.github.loki4j.logback.JavaHttpSender by default.

Yeah, that's correct.

Please clarify if this issue with send delays under Java 11 is reproducible only for ApacheHttpSender or for both ApacheHttpSender and JavaHttpSender.

[czterocyty]

This slowdown was only for JavaHttpSender.

When running on JDK11 with ApacheHttpSender everything worked instantly.

[nehaev]

I tried to reproduce this on Ubuntu 18.04 and OpenJDK 11.0.11, but no luck. Are you sure there's no networking specific involved (e.g. https, dns)? Is you Loki server running on localhost?

[nehaev]

Closing as non-reproducible. Unlikely this has anything to do with Loki4j itself, but feel free to re-open if any details on how to reproduce this appear.