MQTT Device is auto rejecting Cloud to Device messages for first time

willyt150 commented 2 years ago

Hello,

This document specifically calls out that MQTT device communication cannot Reject a Cloud to Device message, and I can confirm that when I try to call the "RejectAsync" method on my DeviceClient, with MQTT, it throws an exception saying it's not supported.

However, we are experiencing for every brand new IoT Device, the first time we send a Cloud to Device message to it, that when the device code connects and attempts to receive the messages, those messages are auto rejected and never received.

Why's it doing that? Is this expected behavior or a bug?

This is occurring on multiple IoT Hub instances

Thanks

Additional Details

IoT Hub instances are all running in South Central US region. (3 instances)
IoT Hubs are all S1 - Standard with 2 IoT Hub units
MQTT connection is made using C#, Microsoft.Azure.Devices.Client v1.41.2.0 NuGet Package DeviceClient.Create with TransportType.Mqtt
I do not have any log traces, and when calling deviceClient.ReceiveAsync(TimeSpan.FromSeconds(5)) there isn't any error, I just receive a null Message.
1. When sending the Cloud to Device message we set the Ack value to DeliveryAcknowledgement.Full, so once the C# code attempts the ReceiveAsync call, we then receive feedback that it was Rejected. FeedbackRecord object contains
  - StatusCode: FeedbackStatusCode.Rejected
  - Description: Message rejected

Once the first message has been auto rejected, every following message is received by the C# client.

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: ba79df4d-c64b-7950-3ae9-6f28ea94409b
Version Independent ID: 7ad0c568-0df2-3be8-f221-0fb846bf575c
Content: Cloud-to-device messages with Azure IoT Hub (.NET)
Content Source: articles/iot-hub/iot-hub-csharp-csharp-c2d.md
Service: iot-hub
GitHub Login: @kgremban
Microsoft Alias: kgremban

RamanathanChinnappan-MSFT commented 2 years ago

@willyt150

Thanks for your feedback! We will investigate and update as appropriate.

willyt150 commented 2 years ago

@RamanathanChinnappan-MSFT thanks for the response, do you have any idea or guess as to when you'll have more information?

We're trying to get these new cloud to device message features released for our projects, however this issue arising is going to limit/prevent us being able to do that and I'm going to need to be able to provide some idea on timeline.

Thanks!

AshokPeddakotla-MSFT commented 2 years ago

@willyt150

Can you please add more info on the issue?

1) Which Region you have created Azure IoT Hub 2) Azure IoT Hub Tier? 3) How did you make the MQTT connection, any specific client used? 4) Any screenshots of the error or the logs would be more helpful to help you further.... 5) Any log traces?

willyt150 commented 2 years ago

@AshokPeddakotla-MSFT Hello, I've updated the original issue w/ Additional Details

Please let me know what else I can provide.

AshokPeddakotla-MSFT commented 2 years ago

@willyt150 We are checking this issue internally with our team. Please stay tuned for further updates. Appreciate your time and patience.

AshokPeddakotla-MSFT commented 2 years ago

@willyt150 Below is the response from our team. Do let us know if that helps.

We have not heard similar issue before, but it will be helpful if you can confirm

whether your IoT Hub is on the new platform (GWV2 = GatewayV2) and whether it's specific to GWV2
You can check this from your IoT Hub's properties as follows

whether it's specific to .NET SDK or you can reproduce the issue with raw MQTT, e.g., Paho MQTT library
Understand Azure IoT Hub MQTT support | Microsoft Learn
This is because .NET SDK is just wrapper of MQTT protocol described above basically

willyt150 commented 2 years ago

@AshokPeddakotla-MSFT Thanks for the response.

All of the IoT Hubs have None listed under Features
I will see what I can do about trying to reproduce the issue with raw MQTT. The issue doesn't occur until my .NET C# code attempts to Receive the message, so honestly it makes me believe it's a bug in the .NET SDK.

I'll keep you posted on my progress/findings

AshokPeddakotla-MSFT commented 2 years ago

@willyt150 Any update/progress on this?

willyt150 commented 2 years ago

@AshokPeddakotla-MSFT unfortunately, I've been given higher priority items than trying to reproduce the issue with raw MQTT (which I'll have to learn/guess at). At this point I do not have an ETA on when I'll be able to get to it.

I was however able to document the exact flow with the .Net C# SDK that results in the issue. (Not sure if anyone on the MS side could attempt the same flow with an already setup raw MQTT system)

IoT Device gets created, in the IoT Hub, via DPS registration from a Symmetric Key Individual enrollment
BEFORE the physical IoT code ever calls DeviceClient.ReceiveAsync a C2D message must be sent
Then have the physical device call DeviceClient.ReceiveAsync, the message gets auto rejected somewhere behind the scenes and the call to ReceiveAsync returns a null Message
All subsequent C2D messages that are sent can be received successfully by the device

The moment the ReceiveAsync call has been made, any C2D messages sent after all work.

AshokPeddakotla-MSFT commented 2 years ago

@willyt150 We have not observed such issues. Sure, we will definitely continue the further troubleshooting once you are available. Incase if you are blocked with this,

Please file a support request @ https://aka.ms/azsupt? If you do not have access to a support plan, please reach out @ AZCommunity@microsoft.com with a link to this Issue as well as your subscription ID and we can help get the support ticket opened for this issue.

Thread URL: # Your Subscription ID: Email Subject: Attn Ashok Peddakotla

Please let me know once you have done the same. Also, for further Technical help, please post to Microsoft Q&A

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

bastyuchenko commented 1 year ago

I've faced with the same issue. I use MQTT and .NET SDK previews/v2 branch. I receive Rejected on backend side.

willyt150 commented 1 year ago

We ended up just building a workaround in our server side code to resend "Rejected" C2D messages up to a configured amount (e.g. 3). After the first rejection, it doesn't auto-reject again.

We keep track of the resend count of the message id ourselves.

It still appears to happen only the very first time for a brand new IoT Device. If we delete an existing IoT device that has been interacting with C2D messages and recreate it, everything works just fine and it's never rejected.

bastyuchenko commented 1 year ago

~~@willyt150 or~~ @AshokPeddakotla-MSFT or @RamanathanChinnappan-MSFT please reopen the issue, I can reproduce this issue.

I use Azure IoT SDK .NET previews/v2 and communicate through MQTT.

Steps for reproduce:

Create Azure IoT Hub
Create a fresh device identity (or it can be an existing device identity, but no session should left from a previous connection in IoT Hub for you ClientId, otherwise messages from this session will be received on device and stored to "offline queue" on SDK side and will be completed)

run a device code (see Console App below)

        X509Certificate2 x509Certificate = Helper.LoadProvisioningCertificate();
        var security = new AuthenticationProviderX509(x509Certificate);
        var registrationId = security.GetRegistrationId();
        IAuthenticationMethod auth = new ClientAuthenticationWithX509Certificate(x509Certificate, registrationId);
        var deviceClient = new IotHubDeviceClient(Helper.IotHubHostName, auth, new IotHubClientOptions(new IotHubClientMqttSettings()));

        deviceClient.ConnectionStatusChangeCallback = (info) =>
        {
            Console.WriteLine($"Device Status {info.Status}; Change reason: {info.ChangeReason}; Recommended action {info.RecommendedAction}.");
        };

        await deviceClient!.OpenAsync(CancellationToken.None);
        await Task.Delay(-1);

run Console App with MessageFeedbackProcessor

async static Task Main(string[] args)
    {
        var iotHub = ConfigurationManager.AppSettings["IoTHubConnectionString"];
        _serviceClient = new IotHubServiceClient(iotHub);

        await ReceiveMessageFeedbacksAsync(CancellationToken.None);
        await Console.Out.WriteLineAsync("feedback listener started");

        await Task.Delay(-1);
    }

    private static async Task ReceiveMessageFeedbacksAsync(CancellationToken token)
    {
        _serviceClient.MessageFeedback.MessageFeedbackProcessor =
        (FeedbackBatch batch) =>
        {
            Console.WriteLine("New Feedback received:");
            Console.WriteLine(JsonConvert.SerializeObject(
                new
                {
                    batch.EnqueuedOnUtc,
                    batch.Records
                }, Formatting.Indented));
            return Task.FromResult(AcknowledgementType.Complete);
        };
        await _serviceClient.MessageFeedback.OpenAsync(token);
    }

run Console App to push c2d messages


    async static Task Main(string[] args)
    {
        _serviceClient = new IotHubServiceClient(ConfigurationManager.AppSettings["IoTHubConnectionString"]);

        while (true)
        {
            await Console.Out.WriteLineAsync("Send message: ");
            Console.ReadKey();
            await SendC2dMessagesAsync(CancellationToken.None);
        }
    }

    private static async Task SendC2dMessagesAsync(CancellationToken cancellationToken)
    {
        var messageId = Guid.NewGuid().ToString("D");
        var correlationId = Guid.NewGuid().ToString("D");
        await Console.Out.WriteLineAsync($"MessageId: {messageId};");
        var message = new OutgoingMessage("test message")
        {
            MessageId = messageId,
            CorrelationId = correlationId,
            // An acknowledgment is sent on delivery success or failure.
            Ack = DeliveryAcknowledgement.Full
        };
        message.Properties.Add("custom-prop-device-id", Helper.DeviceId);

        Console.WriteLine($"Sending C2D message with Id {message.MessageId} to {Helper.DeviceId}.");

        await _serviceClient.Messages.OpenAsync();
        await _serviceClient.Messages.SendAsync(Helper.DeviceId, message, CancellationToken.None);
       Console.WriteLine($"Sent message with Id {message.MessageId} to {Helper.DeviceId}.");
    }

send message from the device (Console App above)

Result:

I have 4 Azure IoT Hubs:

MessageFeedbackProcessor receives "statusCode": "Rejected", "description": "Message rejected" regarding the sent message for three IoT Hub. Features flag for these Azure IoT Hub
MessageFeedbackProcessor receives "statusCode": "Success" regarding the sent message for 1 IoT Hub. Features flag for these Azure IoT Hub

I want to understand is it a bug in Azure IoT Hub or in Azure IoT SDK .NET or in MQTTNet (that is used in Azure IoT .NET SDK).
Is it possible to update Azure IoT Hub Features to GWV2 somehow from my side?
Should I share with you any other information to shed light on the situation? Maybe somehow export schema of IoT Hub and share with you?
Could you upgrade my Azure IoT Hub instance Features to GWV2 to confirm that this upgrade really changes the situation?

willyt150 commented 1 year ago

@willyt150 or @AshokPeddakotla-MSFT please reopen the issue, I can reproduce this issue.

I am unable to reopen the issue, I think it would need to be @AshokPeddakotla-MSFT, @RamanathanChinnappan-MSFT or another collaborator

MicrosoftDocs / azure-docs