[Enhancement request] Circuit breaker to turn off Loki forwarding

[KafkaProServerless]

Hello team,

I would like to reach out for a small enhancement request. It would be great to have a flag, something like (just a suggestion):

<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>some-flaky-loki</url>
            <active>true</active>
        </http>

This active, sendTo, enabled, or amy relevant name would allow some kind of flag to turn off and on the actual forwarding of the log to Liki server.

This is particuluarly helpful to Loki logback users which are using a "flaky, unstable" loki server.

Instead of having issues and issues when not able to send to loki server (and possibly adding load to a loki server trying to recover), this will allow user to just change a logback file to turn the flaf on and off, similar to the famous circuit breaker pattern.

Better if it could be done at runtime, without app restart.

Thank you

[nehaev]

Hi @KafkaProServerless, thanks for reporting this! Have you tried to configure retries for Loki4j and to enable Logback's file appender to write all logs to file in parallel with Loki4j writing to Loki?

Instead of having issues and issues when not able to send to loki server

What issues do you mean in particular?

[KafkaProServerless]

Thank you for your consideration @nehaev

Our use case is simple, in our logback file, we configure two places, destinations, to write logs

1 - into a log file within the same host application is run, something like /logs/application.log

2 - to send the logs over the network, to our internal self hosted Loki instance.

We want both to happen independently from the other, meaning if the Loki server is down, the host machine would still have /logs/application.log, but the aggregation, searches, and alerts from Loki won't be available (expected, since it is down)

The issue arises when Loki server is out. Currently, let's say the Loki server is down in the middle of application writing logs, we will just see a plethora of errors writing to the Loki server (again, expected, since it is down) and we will see network interactions happening between the application and Loki server (despite knowing server is down). -> It would be great if there is a flag to simply disable sending logs to the Loki server at that point

Another case is when Loki server is trying to "catch its breath", having issues, but recovering, in which case, it would be great to still have the ability to turn off the sending to Loki server, or to add an exponential backoff, facilitating Loki server to recover.

This feature, which is just the famous design pattern circuit breaker, would be a nice addition to this project.

[nehaev]

Ok, so if you want to disable the appender from the configuration, please check this SO question.

If you want to fine-tune the retry strategy and exponential backoff, please check maxRetries and maxRetryBackoffMs settings in Loki4j docs.

Circuit breaker is out of the scope for this project.