log4j2-elasticsearch overview

This is a parent project for log4j2 appender plugins capable of pushing logs in batches to Elasticsearch clusters.

Latest released code (1.6.x) is available here.

Project consists of:

log4j2-elasticsearch-core - skeleton provider for conrete implementations
log4j2-elasticsearch-hc - optimized Apache Async HTTP client compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters
(Since 1.6.0) log4j2-elasticsearch-ahc - AsyncHttpClient compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters
log4j2-elasticsearch-jest - Jest HTTP Client compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters
log4j2-elasticsearch2-bulkprocessor - TCP client compatible with 2.x clusters
log4j2-elasticsearch5-bulkprocessor - TCP client compatible with 5.x and 6.x clusters
log4j2-elasticsearch6-bulkprocessor - TCP client compatible with 6.x clusters

Features

Asynchronous log delivery
Batch size and flush interval configuration
Failover (redirect failed batch to alternative target)
JSON message format (user-provided or JacksonJsonLayout by default since 1.3 or Log4j2 JsonLayout)
(Since 1.1) Index rollover (hourly, daily, etc.)
Index template configuration
(1.2) Basic Authentication (XPack Security and Shield support)
HTTPS support (XPack Security and Shield - visit submodules for compatibility matrix)
(1.3) Pooled buffers (lower memory footprint)
Configurable JSON output using JacksonJsonLayout
(1.4) Failover with persistence and retry
Log overflow prevention with backoff policies
log4j2-elasticsearch-hc module - optimized HTTP client
Custom JSON output properties support using VirtualProperty and (since 1.4.3) filters
Pluggable internal logging (since 1.4.3)
(1.5) ILM policy configuration
Configurable Jackson modules support
Component templates configuration
Composable index templates configuration
Service Discovery for HC module
(1.6) Elasticsearch 8.x support (null mapping type)
Data Streams support
Metrics
log4j2-elasticsearch-ahc module - HTTP client with GZIP support

Roadmap

Send Metrics to Elasticsearch
Prep for "batch-core" (or whatever the final name is) module based on common AHC and HC classes
More Elasticsearch API integrations
TRY to maintain compatibility with OpenSearch

Feature Requests welcome!

Usage

Add this snippet to your pom.xml file:

<dependency>
    <groupId>org.appenders.log4j</groupId>
    <artifactId>log4j2-elasticsearch-jest</artifactId>
    <version>1.6.1</version>
</dependency>

Ensure that Log4j2 and Jackson FasterXML jars are added as well - see Dependencies section below

Use simple log4j2.xml configuration:
```
<Appenders>
    <Elasticsearch name="elasticsearchAsyncBatch">
        <IndexName indexName="log4j2" />
        <JacksonJsonLayout />
        <AsyncBatchDelivery>
            <IndexTemplate name="log4j2" path="classpath:indexTemplate.json" />
            <JestHttp serverUris="http://localhost:9200" />
        </AsyncBatchDelivery>
    </Elasticsearch>
</Appenders>
```
or use optimized Apache HC based HTTP client

or new AsyncHttpClient (Netty) based HTTP client

or log4j2.properties

or configure programmatically

NOTE: indexTemplate.json file is not a part of main jars. You have to create it on your own (because only YOU know which mapping you'd like to use). You can find a few basic ones in tests jars and log4j2-elasticsearch-examples.

Start logging directly to Elasticsearch!

Logger log = LogManager.getLogger("Logger that references elasticsearchAsyncBatch")
log.info("Hello, World!");

Logs not arriving? Visit examples and verify your config.

General recommendations

Start simple with jest module. Suitable for smaller loads, up to few thousands of logs per second
Use hc up to 100-200kps (depends on log size and network bandwidth)
Use ahc and GZIP for all of the above and 200kps+ (depends on log size and network bandwidth)

Dependencies

Be aware that Jackson FasterXML, Log4j2, Apache HC, AsyncHttpClient, Netty, Chronicle or JCTools jars (depends on the module you choose) may need to be provided for this library to work. By design, you can choose which jars you'd like to have on your classpath. Please visit mvnrepository for an overview of provided and compile dependencies

In order to fix #56, two new modules were extracted from log4j2-elasticsearch-core:

(1.5+) appenders-logging (compile) available here
(1.5+) appenders-jackson-st (compile) available here

This will not cause any issues if you're using packaging tools with transitive dependencies support (Maven, Gradle, etc.). However, in some cases e.g. if you're managing your jars explicitly, classloaders will complain. Sorry for the inconvenience.

Released to Sonatype OSS repos

Visit submodules' documentation or mvnrepository for XML snippets.

rfoltyns / log4j2-elasticsearch

readme