search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.2k stars 3.5k forks source link

Set `pipeline.buffer.type` to `heap` by default #16353

Open robbavey opened 2 months ago

robbavey commented 2 months ago

In #16054, we introduced a new setting to allow users to set the allocation of memory for certain input plugins to use either direct or heap memory.

We should consider making this the default setting for Logstash 9.0

### Tasks
- [ ] on `main`, when `pipeline.buffer.type` is not set, we should emit a deprecation log guiding to do one of two things before upgrading to 9.x: _either_ (1) set `pipeline.buffer.type: heap` and tune heap size upward in advance of upgrading to 9.x _or_ (2) set `pipeline.buffer.type: direct` to lock in the existing behaviour.
- [ ] on `9.x`, set default for `pipeline.buffer.type` to `heap`
flexitrev commented 1 month ago

Would this be a breaking change if we are only changing the default?

jsvd commented 1 month ago

Any deployment out there consuming from Agent or Beats with a heap usage above 75% (or even 50%) can potentially crash on upgrade as any direct memory allocation from beats/agent/http/tcp inputs would now be done on the heap.