Partial messages merge support

[tomas-bareikis]

We are running applications on AWS Fargate and send logs to New Relic by utilising this Fluentbit plugin. Specifically, we are using the Docker image for AWS Firelens which is defined by this Dockerfile.

New Relic supports storing log messages up to 128000 bytes. However, we have noticed, that long log messages are being split into multiple messages.

This splitting is done by Docker. Docker splits long log messages into 16k chunks. It does mark the messages as partial ones. Reassembling those messages into one can be done by whatever ingests the logs. In this case, it is Fluentbit.

Fluentbit does support merging split log messages into one. It can be achieved by adding the following filter to the configuration:

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    mode                  partial_message

It is possible to mount a config file to the Docker image as stated in the docs, however, it is tricky on AWS Fargate. So, we ended up with building our own custom image with updated configuration.

This is probably a problem that every user running applications on Docker would face when sending logs to New Relic. Would it be possible to include such logs merging feature into the docker image provided by New Relic?

[adomaskizogian]

Our organization is also really looking forward to this feature as we're exactly in the same scenario. We run services on ecs. We produce logs that exceed docker max log entry length. There's no way to concatenate them on new relic ui and does slow down triage.

If it is a matter of updating the fluentbit config only for firelens image, I am willing to put in the work and sumbit a PR.