rebus-org / Rebus.RabbitMq

:bus: RabbitMQ transport for Rebus
https://mookid.dk/category/rebus
Other
63 stars 44 forks source link

.Net Core application freezes on exit #66

Closed MopsiMauser closed 4 years ago

MopsiMauser commented 4 years ago

Hello,

First of all I want to say thanks to this great library and the whole Rebus project. It saved me a lot of time. But I encountered a problem with Rebus.RabbitMq version 6.1 : My application does not exit properly because the thread responsible for AMQP communication blocks the application from exiting (as far as I can see it). I am using Microsofts' hosting extension version 3.1.3 in order to start the application and keep it running. .Net Core Version is 3.1 and the application runs on Windows 10. The application structure looks like this (simplified):

`public class Program { public static void Main(string[] args) { var host = CreateHostBuilder(args).Build(); host.Run(); }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureLogging(ConfigureLogging)
            .ConfigureServices((hostContext, services) =>
            {
                services.AddHostedService<Worker>();
                services.AddRebus(rc =>
                {
                    rc.Transport(t => t.UseRabbitMqAsOneWayClient("amqp://user:bitnami@localhost:5672"))
                    .Logging(l =>
                    {
                        l.Serilog();
                    });
                    return rc;
                });
            });

    private static void ConfigureLogging(HostBuilderContext context, ILoggingBuilder builder)
    {
        builder.ClearProviders();
        var serilogLogger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .ReadFrom.Configuration(context.Configuration)
        .CreateLogger();
        builder.AddSerilog(serilogLogger, true);
        Log.Logger = serilogLogger;
    }
}`

`public class Worker : BackgroundService { private readonly ILogger _logger; private readonly IBus _bus;

    public Worker(ILogger<Worker> logger, IBus bus)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _bus = bus ?? throw new ArgumentNullException(nameof(bus));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation("Publishing message");
            await _bus.Publish(new TextMessage { Text = $"Hello {DateTime.Now}" }).ConfigureAwait(false);
            await Task.Delay(1000, stoppingToken);
        }
    }

    public override Task StopAsync(CancellationToken cancellationToken)
    {
        //dispose IBus manually for testing purpose
        // _bus.Dispose(); //This call causes the application to freeze on exit either called manually or implicitly via ServiceProvider.Dispose() within the IHost
        return base.StopAsync(cancellationToken);
    }
}`

When I stop the application (via Ctrl+c) the described problem appears and the application doesnt stop because 1 thread (apparently the AMQP thread) is still running.

Thanks in advance for your help.

MopsiMauser commented 4 years ago

I have some additional information:

Right after Dispose is called on the IBus instance this line is logged:

2020-04-24T15:16:33.6190183+02:00 [DBG] (ReBus.Publisher/Rebus.RabbitMq.RabbitMqTransport) Initializing new model

and the RabbitMQ web interface signals a blocking connection:

Name User name State SSL / TLS Protocol Channels From client To client
10.0.2.2:64590undefined user blocked โ—‹ AMQP 0-9-1 1 0iB/s 0iB/s

I assume it is a timing problem when a message is published and the application is going to exit nearly at the same time. It seems that a new connection/session is established that causes the freeze/blocking thread.

MopsiMauser commented 4 years ago

I also found it that the connection is already blocked when the IBus object gets instantiated. So it is not a specific problem when the application is about to exit but a general problem using Rebus.RabbitMq with the IHost apllication strucuture i guess. Right after the IBus is instantiated I can see the following status of the connection:

Name User name State SSL / TLS Protocol Channels From client To client
10.0.2.2:60209undefined user blocking โ—‹ AMQP 0-9-1 0 0iB/s 2iB/s

With a hint that the connection will be block on publish.

mookid8000 commented 4 years ago

Interesting! ๐Ÿ™„ Maybe I'll have time to look at it tomorrow. If you come by any more information, which you believe could be useful, then please post it here. ๐Ÿ™‚ Thanks for reporting your findings!

MopsiMauser commented 4 years ago

I have to say sorry for this issue post :-( . I have found the cause of the problem: it is the RabbitMQ server itsself. The server used the wrong virtual disc and had not enough disc space according to RabbitMQ server settings. So the server blocked every connection on publish. And i guess the application freezes on exit because the RabbitMQClient waits for all messages to be published when IBus.Publish has been invoked, but this never happened under these circumstances. After changing the server settings everything worked fine.

mookid8000 commented 4 years ago

Ah, perfect!

I'm so happy to hear that ๐Ÿ˜ƒboth the fact that you figured it out, and also the fact that it wasn't a problem with Rebus after all.