search

rebus-org / Rebus.Autofac

:bus: Autofac container adapter for Rebus
https://mookid.dk/category/rebus
Other
12 stars 11 forks source link

Graceful shutdown? #25

Closed micdah closed 2 years ago

micdah commented 2 years ago

I am not sure if there is an issue per-se, but I am investigating an issue we are seeing sometimes where a message handler is being executed after the root container has been disposed, throwing ObjectDisposedException's.

In my digging I fell upon this interesting issue over on the Rebus main repository: https://github.com/rebus-org/Rebus/issues/976

It sounds very similar and from the looks of it, it was resolved by the RebusBackgroundService here: https://github.com/rebus-org/Rebus.ServiceProvider/blob/master/Rebus.ServiceProvider/ServiceProvider/Internals/RebusBackgroundService.cs

The thing is, we are using Autofac as out IoC layer, on top of ServiceProvider / IServiceCollection. And from the looks of it, I cannot find any similar code in this repository handling the graceful shutdown when the root container is disposed.

Might there be a similar edge case here?

It occurs rather seldom, but often enough to be a source of noise in our logs 🙈

mookid8000 commented 2 years ago

It's definitely possible. What happens is most likely that the following sequence plays out:

  1. Rebus worker receives a message
  2. App shuts down and calls Dispose() on the container
  3. Container disposes its disposable singletons, one of them being Rebus' RebusBus
  4. The Dispose() method of RebusBus blocks to wait for handlers to finish
  5. Rebus worker tries to resolve handlers for its message

If you want to anticipate the error, feel free to

container.GetRequiredService<IBus>().Dispose();

when shutting down before disposing the container.

This will let Rebus finish handling all received messages and stop all workers gracefully.

Rebus' dispose method will ignore any additional calls to it.

If you know how this situation can be avoided with Autofac, I'd love to hear about it 🙂

micdah commented 2 years ago

@mookid8000 sorry for the late response, great suggestion - I ended up doing something very similar for a more graceful shutdown.

I have unfortunately not been able to determine a good hook point within Autofac where such a graceful shutdown could be added, so manually doing hooking it up seems the only solution. Just something to be mindful of.