BeatsTransport _batch_size

eht16 / python-logstash-async

Python logging handler for sending log events asynchronously to Logstash.

MIT License

182 stars 52 forks source link

BeatsTransport _batch_size #93

Closed cbeaujoin-stellar closed 5 months ago

cbeaujoin-stellar commented 7 months ago

I think the batch_size should not be equal to 10 but should be equal to constants.QUEUED_EVENTS_BATCH_SIZE.

I.E if constants.QUEUED_EVENTS_BATCH_SIZE = 1500, it will generate 1500/10=150 transactions and that will take lot of time.

eht16 commented 6 months ago

We could make it configurable more easily.

The higher the batch_size is the more likely will be transmission errors, like timeouts or maybe processing errors on the Logstash side. While the Beats protocol is designed to handle such problems, I'd still keep the default conservatively low.

What about:

making it easier to configure
use a dynamic default value like batch_size = max(50, constants.QUEUED_EVENTS_BATCH_SIZE)?

cbeaujoin-stellar commented 6 months ago

Hi, Yes it sounds a good trade off.

eht16 commented 5 months ago

@cbeaujoin-stellar I implemented the new setting QUEUED_EVENTS_BEATS_BATCH_SIZE with a simple default. I think this is OK as the batch size is configurable and users can set it as they like.