Rebus inside HostBuilder throws exception on dispose

[agerchev]

I am using Rebus using HostBuilder like this:

var hostBuilder = new HostBuilder()
...
.ConfigureServices((context, services) =>
{
    ....

    services.AddRebus(rconfig => rconfig
            .Transport(t => { t.UseOracle(".....", tableName, messageQueueName); })
            .Options((options) =>
            {
                ...
            })
                .Routing(r =>
                          r.TypeBased()
                                .Map<SigningProcessCompleteMessage>(messageQueueName)
                    ));
}); 

host = hostBuilder.Build();

// start rebus
host.Services.UseRebus();

.... 
// On Exit 
host.StopAsync().Wait();
host?.Dispose();

When we try to close the application when pumping and processing messages an exception is thrown in the processing pipeline:

[ERR] Rebus.Workers.ThreadPoolBased.ThreadPoolWorker (Rebus 1 worker 8): Unhandled exception while handling message "SigningProcessCompleteMessage/30ceeaa0-06d9-4093-b1aa-29a26df2075a"
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Rebus.ServiceProvider.NetCoreServiceProviderContainerAdapter.GetHandlers[TMessage](TMessage message, ITransactionContext transactionContext)
   at Rebus.Handlers.InternalHandlersContributor.GetHandlers[TMessage](TMessage message, ITransactionContext transactionContext) in C:\projects-rebus\Rebus\Rebus\Handlers\InternalHandlersContributor.cs:line 39
   at Rebus.Pipeline.Receive.ActivateHandlersStep.GetHandlerInvokers[TMessage](TMessage message, ITransactionContext transactionContext, Message logicalMessage)
   at Rebus.Pipeline.Receive.ActivateHandlersStep.Process(IncomingStepContext context, Func`1 next) in C:\projects-rebus\Rebus\Rebus\Pipeline\Receive\ActivateHandlersStep.cs:line 43
   at Rebus.Pipeline.Receive.HandleRoutingSlipsStep.Process(IncomingStepContext context, Func`1 next) in C:\projects-rebus\Rebus\Rebus\Pipeline\Receive\HandleRoutingSlipsStep.cs:line 42
   at Rebus.Pipeline.Receive.DeserializeIncomingMessageStep.Process(IncomingStepContext context, Func`1 next) in C:\projects-rebus\Rebus\Rebus\Pipeline\Receive\DeserializeIncomingMessageStep.cs:line 34
   at Rebus.Retry.FailFast.FailFastStep.Process(IncomingStepContext context, Func`1 next) in C:\projects-rebus\Rebus\Rebus\Retry\FailFast\FailFastStep.cs:line 41
   at RabbitMQTest.OracleAQRetryStrategy.NoopRetryStrategy.Process(IncomingStepContext context, Func`1 next) in C:\Users\atanas\documents\visual studio 2017\Projects\ConsoleApp1\RabbitMQTest\RebusOracleTest.cs:line 223
   at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ProcessMessage(TransactionContext context, TransportMessage transportMessage) in C:\projects-rebus\Rebus\Rebus\Workers\ThreadPoolBased\ThreadPoolWorker.cs:line 167

The problem seems to be in the process of disposing the container and disposing the bus. If i explicitly dispose the bus everything is normal.

//host.Services.UseRebus();
IBus bus = host.Services.GetRequiredService<IBus>();

//and this one befere container dispose 
bus.Dispose();

[mookid8000]

My guess is that Rebus is not properly disposed, when the service provider container is disposed, as it looks like your Rebus instance is still running and processing messages, even though the hosting container is disposed.

Do you know if the service provider container can automatically dispose the disposables in it?

[agerchev]

I am using the default Ms container. I think that it is disposing the created objects. I saw in the rebus implementation that the workers are stoped when the bus instance is disposed. If the workers are running how can the pending messages be processed when we are disposing the container(to start the disposing of the bus) If the container is in the process of disposing it cannot serve new requests, I think. Am I wrong ?

[agerchev]

@mookid8000 any comment ?

[mookid8000]

@agerchev I'm so sorry! I've completely missed that you wrote in here.

Have you figured something out?

As you correctly noted, this very much looks like an issue that comes from the fact that the service collection refuses to serve any more requests, once the call to Dispose has begun... and then, since Rebus might have just taken a message out of its input queue, it might need to resolve some message handlers, before it can shut down.

I think the solution to this problem could be to hook Rebus up somehow with the lifetime events of the generic host. Do you know if this would be a good way forward?

[agerchev]

@mookid8000 we have created a simple hosted Service, that do the magic with starting and properly stopping the bus. We included it here (for now): https://github.com/agerchev/Rebus.Oracle/blob/master/Rebus.Oracle/ServiceProvider/RebusHostingService.cs

We do not call UseRebus anymore. If you decide to include this or another version, please tell me, so we can remove our fix :)

[Hawxy]

Rebus.ServiceProvider.NetCoreServiceProviderContainerAdapter

@agerchev Could you try using the v5 beta version of this package? The changes I made back in February should result in rebus respecting the lifetime of the container a bit better.

[mookid8000]

As an experiment, I've tried hooking Rebus up with IApplicationLifetime, if it's present in the container.

This way Rebus will stop when the application is stopping, hopefully avoiding trying to resolve any more message handlers, while the container is being disposed.

It's available in Rebus.ServiceProvider 5.0.0-b04, which is on Nuget.org now 🙂