search

rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
12.1k stars 3.9k forks source link

Federation link distribution across cluster nodes #2546

Open michaelklishin opened 8 years ago

michaelklishin commented 8 years ago

See this mailing list thread for the background.

Federation links all placed on a single node create obvious limitations to how many of them can run sustainably in a single cluster. Distributing the links between cluster nodes would not only scale much better but also make federation more predictable operationally, e.g. during upgrades.

hairyhum commented 8 years ago

So idea is to have several exchange downstreams for same exchange (at different upstreams) across cluster?

----------    
| UP 1   |__________________________________
| EX foo |                                  |
----------                                  |
                                            |
----------                                  |
| UP 2   |_______________________           |   
| EX foo |                      |           |
----------                      |           |
                                |           |
----------                      |           |
| UP 3   |____________          |           |
| EX foo |           |          |           |
----------           |          |           |
                     |          |           |
                ----------------------------------
                | DOWN_1   | DOWN_1   | DOWN_3   |
                | EX_F foo | EX_F foo | EX_F foo |
                ----------------------------------
                |            EX foo              |
                ----------------------------------

Because links for different exchanges can be created on different nodes of cluster.

michaelklishin commented 8 years ago

The idea is to distribute links across cluster nodes. This is a consensus problem, so we might want to wait and consider it once we have a better idea of where and how exactly we can use Raft.

hairyhum commented 8 years ago

What I've got from mailing list thread doesn't require consensus and is about manually configuring cluster. But consensus can be used to automatically reconfigure links distribution in case of failure or setup using some criteria.

michaelklishin commented 8 years ago

That specific thread it can be about manual configuration. There were others and the problem is that federation links all live on a single node.

hairyhum commented 8 years ago

I see. Then it would be nice to have consensus for automatic rebalancing. But right now there are still a way to make multiple links to same exchange.

michaelklishin commented 8 years ago

Multiple links to an exchange should be a separate issue. This one is not pressing, too.

dmvieira commented 4 years ago

Do you have news on that? When one node of my cluster falls all federations stop. Upstream still configured but with no federation status on RabbitMQ 3.7.24 with Erlang 21.3.8.14.

I need to remove federation and recreate. Is this related to this problem right?

michaelklishin commented 4 years ago

There is no news. Exchange federation links will be moved to a different node. Queue federation links won't (not sure if this is by design). In both cases, a link would always reside on a single node. This issue is about distributing them for both improved availability and distributing the load. They would still reside on one node each.

dmvieira commented 4 years ago

Strange... I'm observing the same issue related here: https://groups.google.com/forum/#!msg/rabbitmq-users/vnUSK25Hc1k/NlNfddwQAQAJ but with version 3.7.24

michaelklishin commented 4 years ago

We cannot suggest anything without logs and/or steps to reproduce. Our team does not use GitHub as a support forum, that's what rabbitmq-users is for. This issue is about one specific thing we'd like to change in this plugin: distribute links across all the cluster nodes where federation is enabled.