Opt-out of reconnecting on disconnected event

[Drew-Kimberly]

Hello fellow Red Hatter! 😄

My consumer client apps are orchestrated with Kubernetes and hence need to be extremely cognizant of proper graceful shutdown. To handle this, we track a bool in state to acknowledge when the app plans to close the connection and leave a grace period where message processing can finish. If a new message is received while we're still in this grace period, we release it and notify the broker to not redeliver the message to the consumer process that's shutting down, i.e.:

if (isConnectionClosing()) {
        console.log(`Releasing message ${context.message.body.id}`);
        context.delivery.release({undeliverable_here: true});
        return;
}

The issue I'm having is that the broker closes our receiver link upon receiving a {undeliverable_here: true} message release. This in turn results in a disconnected event and rhea's reconnect logic is invoked. Within the disconnected event handler we have access to isConnectionClosing() and therefore can tell rhea to not attempt to reconnect since we know we're in a shutdown grace period and the connection is about to close. My question is- is this possible now or relatively easy to implement?

The other option to consider is to opt-out of rhea's reconnect and handle it myself, tho I'd really like to avoid this if possible.

Please let me know! -Drew

[grs]

What broker are you using? I would not expect a broker to close your link in response to a released message with undeliverable_here=true. It is also not clear why the closing of the link would cause disconnection.

However have you tried setting connection.options.reconnect = false at the point where you start the grace period (or just before you release the message)?

[Drew-Kimberly]

context.connection.options.reconnect = false did the trick, thanks!

As for the broker, we're using an internal MaaS platform which is based off of ActiveMQ. That's interesting that it's not expected behavior, I'll raise this to them today. In case I'm missing something, here is the rhea event debug trace of what happens when a message is released:

Received a SIGINT signal!
  rhea:events [connection-1] Link got event: message +3s
Releasing message 3
  rhea:events [connection-1] Link got event: receiver_close +69ms
  rhea:events [connection-1] Session got event: receiver_close +0ms
  rhea:events [connection-1] Connection got event: receiver_close +0ms
  rhea:events [d8d60da2-b872-e245-86e0-d5712248ef62] Container got event: receiver_close +0ms
  rhea:events [connection-1] Connection got event: disconnected +49ms
  rhea:events [d8d60da2-b872-e245-86e0-d5712248ef62] Container got event: disconnected +0ms
DISCONNECTED
  rhea:events [connection-1] Connection got event: disconnected +185ms
  rhea:events [d8d60da2-b872-e245-86e0-d5712248ef62] Container got event: disconnected +0ms
DISCONNECTED
{ Error: connect ECONNREFUSED xx.x.xx.xxx:5671
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: 'xx.x.xx.xxx',
  port: 5671 }

The part of that trace that seemed peculiar to me was the fact that the disconnected event is being emitted twice in a row. The 2nd occurrence is where context.error is populated with the ECONNREFUSED error.

[furious-pineapple]

Somewhat piggy backing on this issue. Why does releasing a message with {undeliverable_here: true} cause the disconnect event to fire? We had to setup a workaround for releasing messages because we observed that whenever we released a message with {undeliverable_here: true} the client would no longer receive messages. Turns out we just had too resubscribe to the broker whenever we release a message. I'm assuming that's because we unsubscribe from the broker whenever we disconnect. Not sure if this the intended behaviour.

[grs]

There is nothing in rhea that would tie undeliverable_here to disconnecting. It is possible your broker is disconnecting you for some reason or that its handling of that flag is preventing you receiving message until you resubscribe. What broker are you using?