search

logzio / logzio-logback-appender

Logback Appender that ships logs using HTTPs bulk
Apache License 2.0
25 stars 19 forks source link
integration logback-appender

readme

Build Status Maven Central

Logzio logback appender

This appender sends logs to your Logz.io account, using non-blocking threading, bulks, and HTTPS encryption. Please note that this appender requires logback version 1.1.7 and above, and from version 2.0.0, java 11 and up.

Technical Information

This appender uses LogzioSender implementation. All logs are backed up to a local file system before being sent. Once you send a log, it will be enqueued in the queue and 100% non-blocking. There is a background task that will handle the log shipment for you. This jar is an "Uber-Jar" that shades both BigQueue, Gson and Guava to avoid "dependency hell".

Installation from maven

JDK 11 and above:

<dependency>
    <groupId>io.logz.logback</groupId>
    <artifactId>logzio-logback-appender</artifactId>
    <version>2.0.1</version>
</dependency>

JDK 8 and above:

<dependency>
    <groupId>io.logz.logback</groupId>
    <artifactId>logzio-logback-appender</artifactId>
    <version>1.0.29</version>
</dependency>

Logback appender also requires logback classic:

<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.7</version>
</dependency>

logback.xml Example Configuration

<!-- Use debug=true here if you want to see output from the appender itself -->
<!-- Use line=true here if you want to see the line of code that generated this log -->
<configuration>
    <!-- Use shutdownHook so that we can close gracefully and finish the log drain -->
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
    <appender name="LogzioLogbackAppender" class="io.logz.logback.LogzioLogbackAppender">
        <token>yourlogziopersonaltokenfromsettings</token>
        <logzioType>myAwesomeType</logzioType>
        <logzioUrl>https://listener.logz.io:8071</logzioUrl>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
    </appender>
    <root level="debug">
        <!-- IMPORTANT: This line is required -->
        <appender-ref ref="LogzioLogbackAppender"/>
    </root>
</configuration>

Parameters

Parameter Default Explained
token None Your Logz.io token, which can be found under "settings" in your account, If the value begins with $ then the appender looks for an environment variable or system property with the name specified. For example: $LOGZIO_TOKEN will look for environment variable named LOGZIO_TOKEN
logzioType java The log type for that appender, it must not contain spaces
logzioUrl https://listener.logz.io:8071 The url that the appender sends to. If your account is in the EU you must use https://listener-eu.logz.io:8071
drainTimeoutSec 5 How often the appender should drain the queue (in seconds)
socketTimeout 10 1000* The socket timeout during log shipment
connectTimeout 10 1000* The connection timeout during log shipment
addHostname false Optional. If true, then a field named 'hostname' will be added holding the host name of the machine. If from some reason there's no defined hostname, this field won't be added
additionalFields None Optional. Allows to add additional fields to the JSON message sent. The format is "fieldName1=fieldValue1;fieldName2=fieldValue2". You can optionally inject an environment variable value using the following format: "fieldName1=fieldValue1;fieldName2=$ENV_VAR_NAME". In that case, the environment variable should be the only value. In case the environment variable can't be resolved, the field will be omitted.
debug false Print some debug messages to stdout to help to diagnose issues
line false Print the line of code that generated this log
compressRequests false Boolean. true if logs are compressed in gzip format before sending. false if logs are sent uncompressed.
format text Optional. json if the logged message is to be parsed as a JSON (in such a way that each JSON node will be a field in logz.io) or text if the logged message is to be treated as plain text.
exceedMaxSizeAction "cut" String. cut to truncate the message field or drop to drop log that exceed the allowed maximum size for logzio. If the log size exceeding the maximum size allowed after truncating the message field, the log will be dropped.

Parameters for in-memory queue

Parameter Default Explained
inMemoryQueueCapacityBytes 1024 1024 100 The amount of memory(bytes) we are allowed to use for the memory queue. If the value is -1 the sender will not limit the queue size.
inMemoryLogsCountCapacity -1 Number of logs we are allowed to have in the queue before dropping logs. If the value is -1 the sender will not limit the number of logs allowed.
inMemoryQueue false Set to true if the appender uses in memory queue. By default the appender uses disk queue

Parameters for disk queue

Parameter Default Explained
fileSystemFullPercentThreshold 98 The percent of used file system space at which the sender will stop queueing. When we will reach that percentage, the file system in which the queue is stored will drop all new logs until the percentage of used space drops below that threshold. Set to -1 to never stop processing new logs
gcPersistedQueueFilesIntervalSeconds 30 How often the disk queue should clean sent logs from disk
bufferDir(deprecated, use queueDir) System.getProperty("java.io.tmpdir") Where the appender should store the queue
queueDir System.getProperty("java.io.tmpdir") Where the appender should store the queue

Code Example

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LogzioLogbackExample {

    public static void main(String[] args) {
        Logger logger = LoggerFactory.getLogger(LogzioLogbackExample.class);

        logger.info("Testing logz.io!");
        logger.warn("Winter is coming");
    }
}

MDC

Each key value you will add to MDC will be added to each log line as long as the thread alive. No further configuration needed.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

public class LogzioLogbackExample {

    public static void main(String[] args) {
        Logger logger = LoggerFactory.getLogger(LogzioLogbackExample.class);

        MDC.put("Key", "Value");
        logger.info("This log will hold the MDC data as well");
    }
}

Will send a log to Logz.io that looks like this:

{
    "message": "This log will hold the MDC data as well",
    "Key": "Value",
    ... (all other fields you used to get)
}

Marker

Markers are named objects used to enrich log statements, so each log line will be enriched with its own. No further configuration needed.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.Marker;

public class LogzioLogbackExample {

    public static void main(String[] args) {
        Logger logger = LoggerFactory.getLogger(LogzioLogbackExample.class);

        Marker marker = MarkerFactory.getMarker("Fatal");
        logger.error(marker, "This line has a fatal error");
    }
}

Will send a log to Logz.io that looks like this:

{
    "message": "This line has a fatal error",
    "Marker": "Fatal",
    ... (all other fields you used to get)
}

Build and test locally

  1. Clone the repository: 
    git clone https://github.com/logzio/logzio-logback-appender.git
cd logzio-logback-appender
  2. Build and run tests: 
    mvn clean compile
mvn test

Release notes

Contribution