search

loki4j / loki-logback-appender

Fast and lightweight implementation of Logback appender for Grafana Loki
https://loki4j.github.io/loki-logback-appender/
BSD 2-Clause "Simplified" License
314 stars 27 forks source link

Setting `verbose=false` does not affect internal log level #200

Closed patpatpat123 closed 9 months ago

patpatpat123 commented 11 months ago

Hello team,

This is my first post in this repo, and if not anything else, thank you for this project.

This forwarder makes it easy to integrate with grafana cloud, easy to use, and easy to configure, thank you.

I am reaching out with a small enhancement request.

Periodically, in the log, I can see something like this:

04:05:29,055 |-INFO in com.github.loki4j.client.pipeline.AsyncBufferPipeline@14d14731 - >>> Batch #25573c061959ec (DRAIN, 56 records, 1 streams, est. size 21,273 bytes) converted to 21,913 bytes
04:05:29,572 |-INFO in com.github.loki4j.client.pipeline.AsyncBufferPipeline@14d14731 - <<< Batch #25573c061959ec (21,913 bytes): Loki responded with status 204

Please do not get me wrong, especially when starting, those info logs are very useful.

However, as the apps are stabilizing, those logs are becoming a bit of a noise. g Therefore, may I ask if we can change those to something like debug level for instance?

Giving the ability to quite those when apps are running fine, while in the meantime, giving the ability to see them when experiencing issues sending logs to loki backend.

Thank you for your time.

nehaev commented 11 months ago

Hi @patpatpat123, thanks for reporting this!

If you don't want to see such log messages, all you need is to disable a verbose mode:

<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
    ...
    <verbose>false</verbose>
</appender>

Or you can simply delete verbose setting from the config as it's false by default. For more details, please see the docs.

patpatpat123 commented 11 months ago

Thank you @nehaev for the clear answer.

Unfortunately, I still see those logs after applying your suggestion.

Please find attached here the full xml:

<configuration debug="true">

    <springProperty name="name" source="spring.application.name" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss} ${PID} %-5level --- [%thread] [${name},%X{traceId:-},%X{spanId:-}] %logger{36} : %msg%n
            </pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>https://xxx.grafana.net/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>app=${name},host=${HOSTNAME},level=%level</pattern>
                <readMarkers>true</readMarkers>
            </label>
            <message>
                <pattern>%d{yyyy-MM-dd HH:mm:ss} ${PID} %-5level --- [%thread] [${name},%X{traceId:-},%X{spanId:-}] %logger{36} : %msg%n</pattern>
            </message>
            <sortByTime>true</sortByTime>
        </format>
        <verbose>false</verbose>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="LOKI"/>
    </root>

</configuration>

I still see: 

00:48:53,439 |-INFO in com.github.loki4j.client.pipeline.AsyncBufferPipeline@14d14731 - >>> Batch #25e9aac798e018 (DRAIN, 9 records, 1 streams, est. size 2,676 bytes) converted to 2,759 bytes
00:48:53,763 |-INFO in com.github.loki4j.client.pipeline.AsyncBufferPipeline@14d14731 - <<< Batch #25e9aac798e018 (2,759 bytes): Loki responded with status 204

In the console output.

What did I miss?

nehaev commented 11 months ago

Interesting! I've tried to reproduce this in a simple java console app with no luck, verbose flag works as expected.

Loki4j configures internal log level here: https://github.com/loki4j/loki-logback-appender/blob/main/loki-logback-appender/src/main/java/com/github/loki4j/logback/Loki4jAppender.java#L110. Probably another framework/lib you're using overrides this setting.

Could you please dig into this and find out which lib, and how exactly it messed with internal logging level?

patpatpat123 commented 11 months ago

Sure @nehaev ,

Thank you for the follow up.

Just wanted to paste my dependencies here, hoping it can help.

[INFO] com.a.b.c:d:jar:1.1
[INFO] +- io.projectreactor.kafka:reactor-kafka:jar:1.3.22:compile
[INFO] |  +- io.micrometer:micrometer-observation:jar:1.12.0:compile
[INFO] |  |  \- io.micrometer:micrometer-commons:jar:1.12.0:compile
[INFO] |  +- io.projectreactor:reactor-core:jar:3.6.0:compile
[INFO] |  |  \- org.reactivestreams:reactive-streams:jar:1.0.4:compile
[INFO] |  \- org.apache.kafka:kafka-clients:jar:3.6.0:compile
[INFO] |     +- com.github.luben:zstd-jni:jar:1.5.5-1:runtime
[INFO] |     +- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] |     \- org.xerial.snappy:snappy-java:jar:1.1.10.4:runtime
[INFO] +- org.springframework.boot:spring-boot-starter-webflux:jar:3.2.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:3.2.0:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:3.2.0:compile
[INFO] |  |  |  \- org.springframework:spring-context:jar:6.1.1:compile
[INFO] |  |  |     +- org.springframework:spring-aop:jar:6.1.1:compile
[INFO] |  |  |     \- org.springframework:spring-expression:jar:6.1.1:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:3.2.0:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.21.1:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.21.1:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:2.0.9:compile
[INFO] |  |  +- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:2.2:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:3.2.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:3.2.0:compile
[INFO] |  |  \- io.projectreactor.netty:reactor-netty-http:jar:1.1.13:compile
[INFO] |  |     +- io.netty:netty-codec-http:jar:4.1.101.Final:compile
[INFO] |  |     |  +- io.netty:netty-codec:jar:4.1.101.Final:compile
[INFO] |  |     |  \- io.netty:netty-handler:jar:4.1.101.Final:compile
[INFO] |  |     +- io.netty:netty-codec-http2:jar:4.1.101.Final:compile
[INFO] |  |     +- io.netty:netty-resolver-dns:jar:4.1.101.Final:compile
[INFO] |  |     |  \- io.netty:netty-codec-dns:jar:4.1.101.Final:compile
[INFO] |  |     +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.101.Final:compile
[INFO] |  |     |  \- io.netty:netty-resolver-dns-classes-macos:jar:4.1.101.Final:compile
[INFO] |  |     \- io.projectreactor.netty:reactor-netty-core:jar:1.1.13:compile
[INFO] |  |        \- io.netty:netty-handler-proxy:jar:4.1.101.Final:compile
[INFO] |  |           \- io.netty:netty-codec-socks:jar:4.1.101.Final:compile
[INFO] |  +- org.springframework:spring-web:jar:6.1.1:compile
[INFO] |  |  \- org.springframework:spring-beans:jar:6.1.1:compile
[INFO] |  \- org.springframework:spring-webflux:jar:6.1.1:compile
[INFO] +- io.projectreactor:reactor-core-micrometer:jar:1.1.0:compile
[INFO] |  \- io.micrometer:micrometer-core:jar:1.12.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.12:runtime
[INFO] |     \- org.latencyutils:LatencyUtils:jar:2.0.3:runtime
[INFO] +- org.springframework.boot:spring-boot-starter-actuator:jar:3.2.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:3.2.0:compile
[INFO] |  |  \- org.springframework.boot:spring-boot-actuator:jar:3.2.0:compile
[INFO] |  \- io.micrometer:micrometer-jakarta9:jar:1.12.0:compile
[INFO] +- io.opentelemetry:opentelemetry-exporter-otlp:jar:1.31.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-trace:jar:1.31.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-metrics:jar:1.31.0:compile
[INFO] |  |  \- io.opentelemetry:opentelemetry-extension-incubator:jar:1.31.0-alpha:runtime
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-logs:jar:1.31.0:compile
[INFO] |  |  \- io.opentelemetry:opentelemetry-api-events:jar:1.31.0-alpha:runtime
[INFO] |  +- io.opentelemetry:opentelemetry-exporter-otlp-common:jar:1.31.0:runtime
[INFO] |  |  \- io.opentelemetry:opentelemetry-exporter-common:jar:1.31.0:runtime
[INFO] |  +- io.opentelemetry:opentelemetry-exporter-sender-okhttp:jar:1.31.0:runtime
[INFO] |  |  \- com.squareup.okhttp3:okhttp:jar:4.12.0:runtime
[INFO] |  |     +- com.squareup.okio:okio:jar:3.6.0:runtime
[INFO] |  |     |  \- com.squareup.okio:okio-jvm:jar:3.6.0:runtime
[INFO] |  |     |     \- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.9.20:runtime
[INFO] |  |     \- org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.9.20:runtime
[INFO] |  |        +- org.jetbrains.kotlin:kotlin-stdlib:jar:1.9.20:runtime
[INFO] |  |        |  \- org.jetbrains:annotations:jar:13.0:runtime
[INFO] |  |        \- org.jetbrains.kotlin:kotlin-stdlib-jdk7:jar:1.9.20:runtime
[INFO] |  \- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:jar:1.31.0:runtime
[INFO] +- io.micrometer:micrometer-registry-otlp:jar:1.12.0:compile
[INFO] |  \- io.opentelemetry.proto:opentelemetry-proto:jar:1.0.0-alpha:runtime
[INFO] |     \- com.google.protobuf:protobuf-java:jar:3.23.4:runtime
[INFO] +- io.micrometer:micrometer-registry-statsd:jar:1.12.0:compile
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.101.Final:runtime
[INFO] |  |  +- io.netty:netty-common:jar:4.1.101.Final:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.101.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.101.Final:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.101.Final:compile
[INFO] |  |  +- io.netty:netty-transport-native-unix-common:jar:4.1.101.Final:compile
[INFO] |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.101.Final:compile
[INFO] |  \- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.101.Final:compile
[INFO] +- io.micrometer:micrometer-tracing-bridge-otel:jar:1.2.0:compile
[INFO] |  +- io.micrometer:micrometer-tracing:jar:1.2.0:compile
[INFO] |  |  +- io.micrometer:context-propagation:jar:1.1.0:compile
[INFO] |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:2.0.9:compile
[INFO] |  +- io.opentelemetry:opentelemetry-api:jar:1.31.0:compile
[INFO] |  |  \- io.opentelemetry:opentelemetry-context:jar:1.31.0:compile
[INFO] |  +- io.opentelemetry.semconv:opentelemetry-semconv:jar:1.21.0-alpha:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-common:jar:1.31.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk:jar:1.31.0:compile
[INFO] |  +- io.opentelemetry.instrumentation:opentelemetry-instrumentation-api-semconv:jar:1.31.0-alpha:compile
[INFO] |  |  \- io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:jar:1.31.0:compile
[INFO] |  \- io.opentelemetry:opentelemetry-extension-trace-propagators:jar:1.31.0:compile
[INFO] +- org.springframework.cloud:spring-cloud-function-context:jar:4.1.0:compile
[INFO] |  +- net.jodah:typetools:jar:0.6.2:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.0:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-function-core:jar:4.1.0:compile
[INFO] |  +- org.springframework:spring-messaging:jar:6.1.1:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.15.3:compile
[INFO] |     \- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.3:compile
[INFO] +- io.confluent:kafka-json-serializer:jar:7.5.1:compile
[INFO] |  +- io.confluent:common-config:jar:7.5.1:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.15.3:compile
[INFO] |  \- io.confluent:common-utils:jar:7.5.1:compile
[INFO] +- com.github.loki4j:loki-logback-appender:jar:1.5.0-m1:compile
[INFO] |  \- ch.qos.logback:logback-classic:jar:1.4.11:compile
[INFO] |     \- ch.qos.logback:logback-core:jar:1.4.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:3.2.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:3.2.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:3.2.0:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.8.0:test
[INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:4.0.1:test
[INFO] |  |  \- jakarta.activation:jakarta.activation-api:jar:2.1.2:test
[INFO] |  +- net.minidev:json-smart:jar:2.5.0:test
[INFO] |  |  \- net.minidev:accessors-smart:jar:2.5.0:test
[INFO] |  |     \- org.ow2.asm:asm:jar:9.3:test
[INFO] |  +- org.assertj:assertj-core:jar:3.24.2:test
[INFO] |  |  \- net.bytebuddy:byte-buddy:jar:1.14.10:test
[INFO] |  +- org.awaitility:awaitility:jar:4.2.0:test
[INFO] |  +- org.hamcrest:hamcrest:jar:2.2:test
[INFO] |  +- org.mockito:mockito-core:jar:5.7.0:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.14.10:test
[INFO] |  |  \- org.objenesis:objenesis:jar:3.3:test
[INFO] |  +- org.mockito:mockito-junit-jupiter:jar:5.7.0:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.1:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-core:jar:6.1.1:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:6.1.1:compile
[INFO] |  +- org.springframework:spring-test:jar:6.1.1:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.9.1:test
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.10.1:test
[INFO] |  +- org.junit.jupiter:junit-jupiter-api:jar:5.10.1:test
[INFO] |  |  +- org.opentest4j:opentest4j:jar:1.3.0:test
[INFO] |  |  +- org.junit.platform:junit-platform-commons:jar:1.10.1:test
[INFO] |  |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] |  +- org.junit.jupiter:junit-jupiter-params:jar:5.10.1:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.10.1:test
[INFO] |     \- org.junit.platform:junit-platform-engine:jar:1.10.1:test
[INFO] +- com.github.victools:jsonschema-generator:jar:4.33.0:test
[INFO] |  +- com.fasterxml:classmate:jar:1.6.0:test
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.15.3:compile
[INFO] \- com.github.victools:jsonschema-module-jackson:jar:4.33.0:test

I am not setting this var statusListener = new StatusPrinter(verbose ? Status.INFO : Status.WARN); explicitely, trying to find out where this might be set

nehaev commented 9 months ago

Ready to re-open if anybody provides a way how this issue can be reproduced.