search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.49k stars 1.53k forks source link

Update Vector's defaults to holistically solve durability #10281

Open binarylogic opened 2 years ago

binarylogic commented 2 years ago

This is a meta issue to look at Vector's durability defaults at a high level given some of the recent changes and improvements to buffers and end-to-end acknowledgements.

We attempted to solve this in the Durability Brief, but I don't think we landed on the right combination of changes given the in-flight work. Now that we're close to completing that work, we should reassess our defaults in a holistic manner to achieve durability by default.

Proposal

(needs more discussion and consensus, many of the below issues overlap and a final design needs to be discussed)

Open questions & comments

  1. If a downstream service goes down, Vector will retry sending data indefinitely in order to not lose data when/if the service comes back online. This poses a couple of problems:
    1. If the data came from the memory buffer, and end-to-end acknowledgements are enabled, the upstream client will eventually timeout and retry sending that data, creating duplicate data (this is not terrible).
    2. For business continuity, some users will want to failover to a backup service instead of retrying indefinitely.
  2. Can we simplify this if consider the defaults in the context of push and pull-based sources. For example, push-based sources benefit from disk buffers while pull-based do not since they typically handle buffering and durability.
scMarkus commented 10 months ago

Is there any progress going on with this rework? I would be greatly interested in trading memory buffers for disk buffers (#10392).

bruceg commented 10 months ago

No, no progress on most of this. Some of it is planned but deprioritized.

scMarkus commented 9 months ago

Is there a particular part I could help with? I am quite interested in this progressing and would be willing to actively support it.

scMarkus commented 2 months ago

Hi, is there any progress on this one? I would still be very much interested in seeing this progressing. Also I would be happy to actively assist.