Increasing Flume Throughput

[jjbuchan]

When flume was first deployed we could handle 2.5M metrics/minute per node.

If the time comes where we need to increase this throughput, we have a few options that should most likely be followed in this order.

• Increase the number of sink threads.
  • The initial deploy has 10 threads per kafka sink per region.
  • In grafana you should be able to see how many events are being processed per thread
  • This gives a good guide for how many new ones you may need
• Increase the batch size of the sink
  • Larger batches improve throughput while adding latency.
  • The transaction capacity of the channel may need to be increased if you do this
• Increase the available memory for Flume
  • In the run file -Xmx12G is set
  • The byteCapacity for the memory channel defaults at 80% of the -Xmx value.

The Flume User Guide is a good place to start reading about these options - https://flume.apache.org/FlumeUserGuide.html