search

rebus-org / Rebus.RabbitMq

:bus: RabbitMQ transport for Rebus
https://mookid.dk/category/rebus
Other
65 stars 45 forks source link

Notify when connection to the bus is lost/restored #53

Closed mindisk closed 5 years ago

mindisk commented 5 years ago

I have a case where the hosting service needs to be notified when the connection to the bus (which is RabbitMQ) is lost and/or restored. Ideally, I would wish to pass a callback (somehow) to the Rebus which would be called when these events occur. I have trouble finding such mechanism in Rebus. Does Rebus support such mechanism? Perhaps it has has some different approach? If it does, could you share the solution?

Thanks.

mookid8000 commented 5 years ago

As far as I can remember, Rebus' RabbitMQ transport does not currently support this.

I do think it logs a couple of messages though, if it detects a broken connection.

Could you look at your logs to detect this situation?

mindisk commented 5 years ago

It does the log when connection is lost to the bus. However, it is not feasible for the service to monitor the logs and execute some action. Ideally, an event could be raised or a callback could be passed that is called on connection lost and restored.

Will this feature will be considered in the future release?

mookid8000 commented 5 years ago

Will this feature will be considered in the future release?

Sure, I believe that would be a nice addition to the transport.

One thing though: The options builder returned from the call to .UseRabbitMq(..) provides the ability to customize the connection factory – could it maybe be used to do what you want?

mindisk commented 5 years ago

I looked in to this approach a bit. I am not sure if it is possible in a simple manner to customize connection factory to track the single connection state. However, it could be achieved on the IConnection itself, since it able to raise events when connection state changes. Perhaps, you have an idea how to customize connection factory to achieve this? Otherwise, until this feature will be released, I will most likely implement a procedure where a service publishes a message to itself to see if the RabbitMQ is able to deliver it back. In this case I can see if RabbitMQ is alive.

mookid8000 commented 5 years ago

I will most likely implement a procedure where a service publishes a message to itself to see if the RabbitMQ is able to deliver it back. In this case I can see if RabbitMQ is alive.

Sounds like a fairly simply way of detecting whether everything is alive and well.

Regarding detecting it at the driver level: If you customize the connection factory by decorating it, you can attach event listeners to every IConnection returned from it.

If you then attach appropriate listeners, you might be able to get what you want.

I'm closing this issues for now, but feel free to continue the discussion in here – especially if you figure something out, it would be awesome to hear about it. 😎

mindisk commented 4 years ago

Hi, Coming a year later, I tried your suggested approach.
I customized the connection factory and attached event listeners to the IConnection. With that I can detect when connection to the RabbitMQ is lost. However, there is no way to find out when connection is restored with this approach (at least not a easy way). I assume it is because Rebus destroys current connection objects and creates new one.

It would be highly desirable that IBus would contain event handlers such as ConnectionLost and ConnectionRecoverd.

mookid8000 commented 4 years ago

It would be highly desirable that IBus would contain event handlers such as ConnectionLost and ConnectionRecoverd

The problem with that is that the bus does not know or care that there's even anything in the world called "connection".

The only way to provide what you're after is to attach to the connection factory somehow. Isn't it possible to intercept all IModels created and do something with them?

mindisk commented 4 years ago

I actually do that, or at least something similar. I attach a callback to ConnectionShutdown event handler for every IConnection that is being created. Once the ConnectionShutdown is invoked I set the connectionShutDown= true outside of ConnectionFactoryDecorator. Every time ConnectionFactoryDecorator creates new IConnection instance, I check, whether connectionShutDown is true, then I can detect that the connection was restored.

This approach works quite well so far. It is very nice that Rebus allows for such customization. Also, thank you for inspiration. Without https://github.com/rebus-org/Rebus.RabbitMq/issues/53#issuecomment-515758957 I would not have been able to do it.

PS: An idea for future development: Rebus.RabbitMq could implement similar approach as well. .UseRabbitMq(..) could allow to pass a Action that could be invoked when connection is lost or restored.