Open medalliaerlich opened 4 years ago
Yeah I haven't ever needed such high performance from this...
This plugin is quite a thin wrapper over libsystemd ... so I can't immediately suggest any obvious candidates for optimisations.
One possible workaround for your usecase would be to use the docker fluentd logging driver https://docs.docker.com/config/containers/logging/fluentd/ thus avoiding the systemd journal (and this plugin)
Using docker fluentd logging driver disables docker logs
command, which would be very frustrating if the logging system it self encounters issue (for example a network issue, we just ran into a case like it a few days ago). I guess that a reason for some people using to this plugin as an alternative.
Btw this plugin prevents fluentd to use multiple workers. It'd be very nice if and enhancement could be made in the way that enabling multiprocessing.
another thing I could suggest is trying fluent bit https://fluentbit.io/documentation/0.12/input/systemd.html
I don’t know ... but perhaps that implementation is more performant...
I am unlikely to work on this myself at the moment... but I am happy to review and merge anything that comes in ...
Hi I'm using this plugin to get logs from journal and then save them to files splitting by container name. The configuration I'm using looks like this:
With this setting I'm only getting a throughput of ~1000 lines per second while, according to https://docs.fluentd.org/deployment/performance-tuning-single-process, FluentD should be able to run 5000 lines per second. A few additional details:
I'm running FluentD inside a Docker container with 4Gb and 4096 CPU shares
Tried with local storage as well as shared storage
Tried removing the file output and using only with metrics as out