vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.31k stars 1.61k forks source link

Remote state persistence #9605

Open binarylogic opened 3 years ago

binarylogic commented 3 years ago

After opening #9604, I think we should consider the ability to store Vector's state remotely based on configuration. This could be an external blob store, an automatic feature of our enterprise offering, etc. This would simplify things like #9604, allow users to avoid disks altogether, and make it more obvious to users how state is persisted and recovered.

jszwedko commented 3 years ago

Doing this with disk buffers would essentially make them external buffers too, no?

binarylogic commented 3 years ago

This is more about storing Vector state, like checkpoints.