[Bug Report] SendEventAsync() hangs forever when the DeviceClient is Closed/Disposed while ConnectionStateChangedHandler is running

[Werner-Prbk]

Context

• OS, version, SKU and CPU architecture used: Windows 11 Desktop x64, linux arm32
• Application's .NET Target Framework : dotnet7.0, dotnet8.0
• Device: Laptop, Embedded Device arm32
• SDK version used: 1.42.0, 1.42.2, 1.42.3

Description of the issue

SendEventAsync() never returns when the DeviceClient is Closed and Disposed while the ConnectionStateChangedHandler is running. The following example enforces this behavior and leads to the deadlock with all v1.42.x. Previous versions (tested with v1.41.3) handle this condition as expected.

It seems that CloseAsync() has changed in behavior: 1.41.3 --> CloseAsync() blocks until the ConnectionStateChangedHandler has finished 1.42.3 --> CloseAsync() does not block

Years ago I reported a similar bug, but I think the underlying problem is different now #2186.

I also think that this problem can occur in the ReconnectionSample.

Code sample exhibiting the issue

Start with internet connection, then disconnect from internet when Cut internet and press enter ... is printed.

internal class Program
{
    private const string ConnectionString = "HostName=****.azure-devices.net;DeviceId=device01;SharedAccessKey=******";
    private static DeviceClient _deviceClient;
    private static ManualResetEventSlim _mre = new(false);

    static void ConnectionStateChangedHandler(ConnectionStatus status, ConnectionStatusChangeReason reason)
    {
        Console.WriteLine($"IotHub connection status changed to {status} because of {reason}.");

        if (status == ConnectionStatus.Disconnected)
        {
            Console.WriteLine("Set MRE");
            _mre.Set();
        // ensure other thread executes Close and Dispose
            Thread.Sleep(10000);
            Console.WriteLine("Exit ConnectionStateChangedHandler");
        }
    }

    static async Task Main(string[] args)
    {
        _deviceClient = DeviceClient.CreateFromConnectionString(ConnectionString, TransportType.Mqtt_Tcp_Only);
        _deviceClient.SetRetryPolicy(
             new ExponentialBackoff(6,TimeSpan.FromSeconds(1),TimeSpan.FromSeconds(10),TimeSpan.FromSeconds(1)));
        _deviceClient.SetConnectionStatusChangesHandler(ConnectionStateChangedHandler);
        await _deviceClient.OpenAsync();

        Console.WriteLine("SendEventAsync...");
        await _deviceClient.SendEventAsync(new Message(new byte[]{ 0x01 }));
        Console.WriteLine("SendEventAsync OK");

        Console.WriteLine("Cut internet and press enter ...");
        Console.ReadLine();

        var t = Task.Run(async () => {
            try
            {
                Console.WriteLine("SendEventAsync...");
                await _deviceClient.SendEventAsync(new Message(new byte[]{ 0x01 }));
                Console.WriteLine("SendEventAsync OK");
            }
            catch (Exception ex)
            {
        // Never reached
                Console.WriteLine("SendEventAsync was aborted with exception as expected");
            } });

        _mre.Wait();
        _deviceClient.SetConnectionStatusChangesHandler(null);
        Console.WriteLine("Close DeviceClient");
        await _deviceClient.CloseAsync();
        Console.WriteLine("Dispose DeviceClient");
        _deviceClient.Dispose();
        Console.WriteLine("Dispose DeviceClient Finished");

        await t;
    }
}

Console log

IotHub connection status changed to Connected because of Connection_Ok.
SendEventAsync...
SendEventAsync OK
Cut internet and press enter ...

SendEventAsync...
IotHub connection status changed to Disconnected_Retrying because of Communication_Error.
IotHub connection status changed to Disconnected because of Retry_Expired.
Set MRE
Close DeviceClient
Dispose DeviceClient
Dispose DeviceClient Finished
Exit ConnectionStateChangedHandler

Traces

iotSdk.log