logstash-plugins / logstash-output-file

Apache License 2.0
23 stars 53 forks source link

add write buffer to improve performance #72

Closed davies closed 4 years ago

davies commented 6 years ago

When writing logs into a network file system (for example, JuiceFS), current plugin is slow because:

1) checking the file exists or not for every chunks of logs. Checking files (multiple lookup operations) is much expensive than local file system,

2) the default write buffer in JRuby is too small (4K), which means the background flush is useless.

When logging comes from UDS then is written into network file system, it's easy to see some logs get lost due to high load of logstash.

This PR address these issues by introducing another write buffer into IOWrite (up to 1MB by default), It also check the files exists or not only before the buffered logs will be written into files. So it will only check once in two seconds with default flush_interval=2.

We benchmark this by having a Python script to generate logging and forward them to logstash using gelf (UDP), saw that the dropped logs are reduced significantly (the other bottleneck is the UDP itself).

colinsurprenant commented 6 years ago

Hello @davies - thanks for your PR. Before reviewing this, could you please sign the CLA? Once this is done, could you please explain the problem you are solving, and if performance related also provide before/after benchmarks?

davies commented 6 years ago

@colinsurprenant We have signed CLA (as Juicedata)