search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.66k stars 1.56k forks source link

Error on boot if Vector is configured to need a disk but one is not available #9604

Open binarylogic opened 3 years ago

binarylogic commented 3 years ago

Vector is often deployed as an aggregator without a disk. Many users do not want a disk due to:

  1. The increased cost.
  2. The complexity disks add to rate limit management in the cloud (AWS IOPs rate limits).
  3. Scars from using disk buffers with previous tools. We hear stories about disk buffers taking down aggregation clusters, etc, and companies are adamant about not using them (external buffers instead).

To ensure Vector is easy to work with, we should error on boot if Vector is configured to require a disk but does not have one. Basically:

  1. Any source the checkpoints.
  2. Disk buffers are enabled.
  3. Anything else I'm missing.

For point 1. I think our enterprise product can offload persistence of state like that, but we aren't there yet. We might even consider the ability to configure how state is persisted and then use that knowledge to enforce the boot-time error proposed here.

Finally, I think this can be grouped into a larger project with issues like #7340

jszwedko commented 3 years ago

I think Vector actually already fails to boot if started with an unwritable data directory if there are any sources that rely on it.

We could (maybe?) have the Helm chart validate this before deployment though.

spencergilbert commented 3 years ago

We could (maybe?) have the Helm chart validate this before deployment though.

👍