Open yogevyuval opened 3 years ago
Could you share your pipeline config? What is the output plugin? Do you see the same problem in 7.12?
Could you share your pipeline config? What is the output plugin? Do you see the same problem in 7.12?
The output plugin was file output plugin, and we havent tried it in 7.12. Unfortunately I don't have the exact pipeline but it was very simple (read from s3, write to file)
Trying to read a 15M gzipped file (200M uncompressed) shows a strange behaviour, with logstash 7.9.2. I tried reading plain files and it doesnt seem to have a difference.
After looking at the debug logs it seems that the download part is really fast, and the local processing is very slow.
I tried looking into the source code of the plugin, removing some of the code to be able to pinpoint the problem. Eventually I removed almost every line in the
process_local_log
function, keeping only the codec (which is plain by default) decoding, and thequeue << event
line. It seems that this is the line that is taking most of the time.Any idea what could be the cause of this? This makes the plugin almost unusable in use cases with large volumes for example flow logs forwarding