search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.44k stars 4.43k forks source link

streaming: Expand Streaming support to all blocking queries #6366

Open banks opened 5 years ago

banks commented 5 years ago

This is a placeholder issue that will be modified over time.

We are currently working on a new internal architecture to power blocking queries that is significantly more efficient. One benchmark based on an extreme but real customer workload showed the following levels of improvement for servers:

Steady CPU: 60% -> <1%, a 60x+ reduction in CPU Memory: 420MB -> 20MB (minus 80MB baseline), a 21x reduction in Memory Outbound Bandwidth: 2.2Gbps -> 0.5Mbps (mins 500k baseline), a 4,400x reduction in network throughput

The public API for now remains the same and this is a purely internal (between client and server agents) change.

The new architecture has been intentionally designed to roll out incrementally to make it more practical and we are closing in on the first point at which we can release a major part of Consul being powered by this.

The first use-case will be the biggest impact one for many workloads - service health watching. Once that is rolled out, this issue is a placeholder for the iterative process of rolling out the same architecture to cover eventually all other blocking APIs.

The proposed order to roll them out in terms of impact is:

Until we have completed and are ready to ship the first use-case this is a placeholder, but once we've done that and shown the patter is successful, the plan is to update this issue with a "guide" walking through the steps needed to upgrade a given API endpoint from the above list to use streaming. From there we can make individual tasks and hopefully burn them down with help from the community until all of Consul's blocking mechanisms are ported to the new streaming mechanism.

More info to come here once we are ready for that.

CC @Aestek @pierresouchay FYI.

pierresouchay commented 5 years ago

Great news, tell us if we can help

banks commented 5 years ago

tell us if we can help

Basically watch this issue and once we have work to do it will be here! Thanks!

ChrisMcKenzie commented 4 years ago

Any update on this feature, would love to contribute if help is needed,

helight commented 4 years ago

we look for this feature for a long time. we want to use this feature ASAP. thanks guys for your great work.

jkirschner-hashicorp commented 3 years ago

Hi all - a quick update on our progress here.

We've made significant progress on implementing streaming for the service health watching use case (see Consul 1.9 release blog post). We've also created some high-level developer-focused documents which can help guide the addition of more streaming topics in the future.

There's still work to be done to polish the service health watching use case and to improve the developer docs for streaming.

At this time, we are not yet in a state where we are ready to support or accept community contributions to implement streaming for other endpoints.

dnlopes commented 2 years ago

Any news on this topic? It has been over an year since the last update.

I'm particurarly interested in the KV endpoint change.

seanmacisaac commented 1 year ago

Another ping on this issue. We could use streaming for the KV endpoints and at some point of scale will likely need to move away from consul if it isn't supported.

dnlopes commented 1 year ago

Another ping on this issue. We could use streaming for the KV endpoints and at some point of scale will likely need to move away from consul if it isn't supported.

Yes, we have used consul-replicate in the past but it does not scale for our use case. A streaming solution that only outputs the deltas would be way more efficient.

seanmacisaac commented 9 months ago

Any progress here? Are you looking for open source contributions?