search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.26k stars 4.41k forks source link

Prepared queries that include results from multiple DCs #2803

Open ross opened 7 years ago

ross commented 7 years ago

We have a use case that we're trying to accommodate with prepared queries, but can't quite get to work. We're trying to setup puppet SRV records, https://docs.puppet.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records.

We can get a prepared query setup that almost does what we're looking for, but not quite. The catch is that we have multiple "well connected" datacenters in a region and want to return all of the puppetservers across those sites, not just the ones in the current DC. This would be analogous AZs within an AWS region.

The Failover functionality is almost what we want, but not quite. We want it to return things from a list of datacenters always, not just as a failover.

It seems slightly related to https://github.com/hashicorp/consul/issues/671, and https://github.com/hashicorp/consul/issues/252 would likely be useful as well, but isn't 100% required.

Anyway, I'm happy to open an issue in the repo to discuss, but I wanted to reach out and check to make sure that I'm not overlooking something or failing to find a previous issue about this.

Moving to an issue from the mailing list.

/cc @slackpad

jjathman commented 4 years ago

We are looking for this feature as well. Any chance on reconsidering the priority of this?

Our use case is where we have a single process doing something like reading a file and generating requests to internal services based on that file. We would like to evenly distribute the load across multiple datacenters if possible rather than having all of the load be handled by the datacenter where the process originates.

blake commented 4 years ago

@jjathman Would simply returning a list of healthy service endpoints from all federated DCs satisfy your use case?

I'm curious what an ideal solution looks like from your perspective, and whether you require something that is not captured in @ross's comment.

jjathman commented 4 years ago

@blake I think @ross’s comments match what we want. We have two Consul data centers that are in the same physical DC. We would love to be able to specify the two Consul data centers in the prepared query so we only use the data centers that are physically near. However using all federated data centers would be better than nothing so it that is a lot easier that works too. Thanks.