How can I use different layouts with HCHttp client , other than JacksonJsonLayout.

J-Man13 commented 4 years ago

Good day, sir I am having a small issue here , could really use your advices or recommendations.

Currently I'm trying to append logs into elastic using Elasticsearch appender configuration with Jest client:

`

You might be wondering , why would i use PatternLayout in the elastic appender , but the main reason for that is because I am using custom messages whenever I'm logging to escape log4j2 inserted metadata in the log message. Everything is working just fine with such configuration for elastic appender. A little bit later I decided to use HCHttp client instead of Jest client because it was mentioned in documentation , that it is more optimized. Here is configuration of appender:

`

` So , this configuration would not wrok , throwing java.lang.UnsupportedOperationException: Use ItemSource based API instead. Here is the my app link , if you wonder about what I am trying to achieve : https://github.com/jafarjafarov/actuatorSwaggerCRUDSample. The main idea is to allow the exposure of different java objects in elastic in json format during some flow execution. Thanks in advance

rfoltyns commented 4 years ago

If you'd like to use HCHttp to get the optimizations, JacksonJsonLayout is the only out-of-the-box layout that you can use. Otherwise, you have to implement your own ItemSourceLayout.

ItemSource API is a part of the optimization here - in this case, it provides a bunch of wrapped, resizable buffers for serialized data (see Object Pooling). Whole HCHttp is based on it, so PatternLayout will not work here, because it serializes to String, not to ItemSource.

If your logs are already in JSON format, you can use messageOnly flag to send the log as-is with following configuration:

<Elasticsearch messageOnly="true" >
    <JacksonJsonLayout>
        <PooledItemSourceFactory itemSizeInBytes="1024" initialPoolSize="20000" />
    </JacksonJsonLayout>
    <AsyncBatchDelivery batchSize="5000" deliveryInterval="1000" >
        ...
        <HCHttp serverUris="...">
            ...
            <PooledItemSourceFactory itemSizeInBytes="5120000" initialPoolSize="4" />
            ...
        </HCHttp>
    </AsyncBatchDelivery>
</Elasticsearch>

Otherwise, you can override LogEvent and Message serializers with JacksonMixIn (see docs) to get desired effect.

With HCHttp, PooledItemSourceFactory config is crucial here. It tells JacksonJsonLayout to use specified factory to serialize logs. Factory in the layout defines buffers for logs (1024 bytes in this case, adjust for your logs accordingly). Factory in HCHttp defines buffers for batch requests.

batch itemSizeInBytes = layout itemSizeInBytes * batchSize

layout layoutPoolSize = batch initialPoolSize * batchSize

rfoltyns commented 3 years ago

@jafarjafarov Did the suggestion above help you solve your issue? Is there anything else you'd like me to address in this scope?

rfoltyns / log4j2-elasticsearch

How can I use different layouts with HCHttp client , other than JacksonJsonLayout. #53