search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
124 stars 134 forks source link

Implement the key-value state store in V2 control protocol #2178

Open blakerouse opened 1 year ago

blakerouse commented 1 year ago

The key-value store that was added to the v2 control protocol needs to be implemented. The requirements are:

cmacknz commented 1 year ago

Do you have a concrete use case for this in mind? I hesitate to prioritize this unless we have a clear use case for it. This is a lot of complexity to bring in, I would want to make sure it is worth it.

blakerouse commented 1 year ago

This was added to the V2 architecture design and already exists in the https://github.com/elastic/elastic-agent-client/blob/main/elastic-agent-client.proto#L491.

The biggest reason to add this was for helping input developers and getting more input developers to work with the elastic-agent. The idea is it removes the need for input developers to have to worry about state management for the life of there running unit. They can use the key-value store as there state store without having to worry about writing state on the filesystem and leave that to the elastic-agent to worry about.

I agree its not something that we need to prioritize but it would be great to add when time allows. Many use cases could be used for the store:

This could also be extended to have different backends based on the environment the Elastic Agent is running. On Kubernetes we need a PVC to store the state of the elastic-agent, but if we offered this key-value store and all inputs used it we could remove the need for a PVC and store the key-value in kubernetes secrets, or talk directly to etcd if they want to enable it. Some K8s environments will not have PVC's so this gets us around that issue with this implemented, with different backends ofcourse.