RawRabbit Connection.Dispose Hangs

[mgw854]

I'm running some tests on an application that uses RawRabbit to ship informational messages. Very infrequently, we run into the case where RabbitMQ is just so overwhelmed that it can't give us back a publisher confirmation for the message. In that scenario, we want to try again a few times, and then eventually drop the message (note, however, that if the message does make it into RabbitMQ, it should stay until it is popped off the queue).

To simulate this, I can run this command inside a Docker container running RabbitMQ:

rabbitmqctl set_vm_memory_high_watermark 0

RabbitMQ will not accept any messages coming in, even though you can connect to it. This works fine, and it behaves normally (well, throws PublishConfirmException) until I try to stop the application. I've traced this with a debug build of RawRabbit (latest), and see that it blocks indefinitely inside of ChannelFactory's Dispose method, specifically this line:

Connection?.Dispose();

I can still see in the flat file logs events being emitted:

[16:28:40 DBG] Channel pool currently has 1 channels open and a total workload of 0

Our configuration:

      var cfg = new RawRabbitConfiguration
      {
        Username = username,
        Password = password,
        VirtualHost = virtualHost,
        Port = port,
        Hostnames = { host },
        RequestTimeout = TimeSpan.FromSeconds(timeout),
        PublishConfirmTimeout = TimeSpan.FromSeconds(timeout),
        PersistentDeliveryMode = true,
        TopologyRecovery = true,
        AutoCloseConnection = false,
        AutomaticRecovery = true,
        Exchange = new GeneralExchangeConfiguration
        {
          AutoDelete = false,
          Durable = true,
          Type = RawRabbit.Configuration.Exchange.ExchangeType.Direct
        },
        Queue = new GeneralQueueConfiguration
        {
          AutoDelete = false,
          Durable = true,
          Exclusive = false
        },
        RecoveryInterval = TimeSpan.FromSeconds(timeout),
        GracefulShutdown = TimeSpan.FromSeconds(timeout),
        RouteWithGlobalId = true
      };

      TimeoutPolicy timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromMinutes(1), TimeoutStrategy.Optimistic);
      RetryPolicy retryPolicy = Policy.Handle<BrokerUnreachableException>()
        .WaitAndRetryAsync(new[] { TimeSpan.FromSeconds(2.5), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(10) });
      PolicyWrap connectionPolicy = timeoutPolicy.WrapAsync(retryPolicy);

      PolicyOptions policyOptions = new PolicyOptions
      {
        PolicyAction = c => c.UsePolicy(Policy.NoOpAsync()),
        ConnectionPolicies = new ConnectionPolicies
        {
          Connect = connectionPolicy
        }
      };

      RawRabbit.Instantiation.Disposable.BusClient client = RawRabbitFactory.CreateSingleton(new RawRabbitOptions()
      {
        ClientConfiguration = cfg,
        DependencyInjection = ioc =>
        {
          ioc.AddSingleton<ISerializer, CustomJSONSerializer>();
        },
        Plugins = p =>
        {
          p.UsePolly(policyOptions);
        }
      });

[houseofcat]

Correct me if I am mistaken, there is zero memory available for the server to work with and Connection.Dispose throws an exception?

If thats the case, the server needs to have some memory to receive the events from client.

RabbitMQ and Erlgang purposely run on a system where it will use every available resource if you let. Such systems do not hold your hand like an operating system would assist in situations like this.

[mgw854]

The remote RabbitMQ server believes that it has no memory available, yes (it's crossed the high water mark). We know that's an undesired scenario, but we hit it once accidentally and want to make sure that our applications can gracefully recover from this situation in the future in case it happens again.

Connection.Dispose doesn't throw an exception, it just hangs forever.

[houseofcat]

Test a regular RabbitMQ connection Dispose().

If it freezes still there isn't much RawRabbit can do no?