logstash-plugins / logstash-input-beats

Apache License 2.0
87 stars 81 forks source link

Logs a warn message on max order overshoot #493

Closed andsel closed 9 months ago

andsel commented 9 months ago

Release notes

Added logging the best Netty's maxOrder setting when big payload trigger an unpooled allocation

What does this PR do?

Print a log line with the preferred Netty's maxOrder to accomodate the buffer size that exceeds the actual chunk size that would trigger an unpooled allocation. It prints the log just one time across all plugin instances for each maxOrder required. So if plugin instance A reaches the limit of 4Mb (aka maxOrder 9) with a desired payload size of 7Mb it will report line about to use maxOrder 10. After this log line is printed, any other plugin instance wouldn't report it again, unless it needs an even bigger maxOrder not yet reported. This PR also introduces a test spy of the Logger class to test the behaviour.

Why is it important/What is the impact to the user?

Helps the user to set the right maxOrder for his message flow to avoid slow unpooled allocation.

Checklist

Author's Checklist

How to test this PR locally

It needs the test tool present in branch ingest_all_hands_ams contained in PR https://github.com/elastic/logstash/pull/15151.

  1. checkout Logstash branch from https://github.com/elastic/logstash/pull/15151
  2. checkout this PR branch
  3. configure Logstash to use the checkouted Beats PR, after vendored it (./gradlew vendor)
    # in Gemfile add:
    gem "logstash-input-beats", :path => "/Users/andrea/workspace/logstash_plugins/logstash-input-beats"
    # install it
    bin/logstash-plugin install --no-verify
  4. eventually configure log (config/(log4j2.properties) add:
    logger.beats_input.name = org.logstash.beats
    logger.beats_input.level = info
  5. run logstash with pipeline like:
    input {
    beats {
    port => 3333
    }
    }
    output {
    sink {}
    }  
  6. run the benchmark tool to create payloads > 4MB, from ingest_all_hands_ams run:
    bundle exec ruby -J-Xmx16g -J-XX:-MaxFDLimit benchmark_client.rb --test=beats --msg_sizes=2500 -a yes
  7. verify in the logs of Logstash, it report the preferred maxOrder of 10, like:
    [2024-02-14T14:18:07,863][WARN ][org.logstash.beats.V2Batch][beats_input][32f207bacda794ed2bf54ea29a1645a3dbb680398856e5cabef16bde58104d8d][defaultEventExecutorGroup-4-5]
    Got a batch size of 4194342 bytes, while this instance expects batches up to 4194304, please bump maxOrder to 10.

Related issues

Use cases

Screenshots

Logs