rabbitmq / rabbitmq-federation

RabbitMQ Federation plugin
https://www.rabbitmq.com/
Other
40 stars 21 forks source link

Improve logging on unexpected behaviour during partial partitions #53

Closed dcorbacho closed 7 years ago

dcorbacho commented 7 years ago

During partial network partitions on the downstream cluster, nodes on different sides of the partition might connect simultaneously to the upstream. That causes upstream exchanges to be recreated with different names (A/B) and links trying to add bindings to exchanges that don't exist anymore. That causes crashes like the one below and restarts the link.

Apparently, the best way to recover the link is to allow it to restart, but the crash can be captured and replaced by a more informative log message.

=SUPERVISOR REPORT==== 19-Jan-2017::15:34:44 ===
     Supervisor: {<0.298.0>,rabbit_federation_link_sup}
     Context:    child_terminated
     Reason:     {{shutdown,
                      {server_initiated_close,404,
                          <<"NOT_FOUND - no exchange 'federation: test_5 -> rabbit@ubuntu-c3 B' in vhost '/'">>}},
                  {gen_server,call,
                      [<0.671.0>,
                       {call,
                           {'exchange.bind',0,
                               <<"federation: test_5 -> rabbit@ubuntu-c3 B">>,
                               <<"test_5">>,<<"test_5">>,false,
                               [{<<"x-bound-from">>,array,
                                 [{table,
                                      [{<<"cluster-name">>,longstr,
                                        <<"rabbit@ubuntu-c3">>},
                                       {<<"exchange">>,longstr,
                                        <<"/:test_5 B">>},
                                       {<<"hops">>,short,1}]}]}]},
                           none,<0.299.0>},
                       infinity]}}
     Offender:   [{pid,<0.299.0>},
                  {name,
                      {upstream,
                          [<<"amqp://192.168.56.108">>],
                          <<"test_5">>,<<"test_5">>,1000,1,5,none,none,false,
                          'on-confirm',none,<<"ubuntu-c4">>,false}},
                  {mfargs,
                      {rabbit_federation_exchange_link,start_link,
                          [{{upstream,
                                [<<"amqp://192.168.56.108">>],
                                <<"test_5">>,<<"test_5">>,1000,1,5,none,none,
                                false,'on-confirm',none,<<"ubuntu-c4">>,false},
                            {resource,<<"/">>,exchange,<<"test_5">>}}]}},
                  {restart_type,{permanent,5}},
                  {shutdown,30000},
                  {child_type,worker}]