search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.25k stars 4.59k forks source link

Azure IoT Hub Generates Duplicate Events #22905

Closed nainesh72 closed 3 years ago

nainesh72 commented 3 years ago

Describe the bug I get a Duplicate Event. In Azure Event Subscription and also Azure Function which is trigger from Azure Event Hub

Expected behavior I get Rendomany Duplicate Message on Cloud, where my edge device sends only one message

Actual behavior (include Exception or Stack Trace) It behavior randomly on any patten

To Reproduce Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

1. 2.

Environment:

jsquire commented 3 years ago

Hi @nainesh72. Thank you for your feedback and we regret that you're experiencing difficulties.

What you're describing is expected behavior, as both IoT Hub and Event Hubs have an at-least-once delivery guarantee. Your application should expect occasional duplicates; it is highly recommended to ensure that your processing is resilient to event duplication in whatever way is appropriate for your application scenarios.

For more information, see:

ghost commented 3 years ago

Hi @nainesh72. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

nainesh72 commented 3 years ago

“/unresolve” I some investigation on this issue that from the device iot hub sends the duplicate message because when i check the IoT hub-enqueue time in system property it will 1-second different with seme message, whereas my message time is 30 second i don't know why this happens

nainesh72 commented 3 years ago

Hi i send detail of duplicate messages which I receive on the cloud hub

2021-07-27T19:29:02.651 [Information] {"properties": {"sequenceNumber": "454"},"systemProperties": {"iothub-connection-device-id": "EdgeDeviceL1","iothub-connection-module-id": "TestModule1","iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id": "637629209519885184","iothub-enqueuedtime": "2021-07-27T19:29:02.214Z","iothub-message-source": "Telemetry"},"body": "eyJkdF91dGMiOiIyMDIxLTA3LTI3VDE5OjI5OjAyLjIwMTg5NDdaIiwic3RhdGlvbklEIjoiRWRnZURldmljZTFMIiwibTF0dCI6MTUwLCJtMXB0Ijo3MywibTFmdCI6MTU1MywibTFzZHAiOjE2LCJtMW9lZXAiOjcwLCJtMXRocnAiOjUzLCJtMWF2YXAiOjIwLCJtMXF1YXAiOjM0LCJtMXN0cyI6MywibTFhX3N0cyI6MiwibTFiX3N0cyI6MiwibTF0b3RiYmxtc3VtIjoxMjcyLCJtMnR0IjozMSwibTJwdCI6NzQsIm0yZnQiOjI1MjAsIm0yc2RwIjoyLCJtMm9lZXAiOjk5LCJtMnRocnAiOjUzLCJtMmF2YXAiOjI4LCJtMnF1YXAiOjMwLCJtMnN0cyI6MSwibTJhX3N0cyI6MiwibTJiX3N0cyI6MiwibTJ0b3RiYmxtc3VtIjoyNzYwLCJtM3R0IjoxMjEsIm0zcHQiOjQ5LCJtM2Z0IjozMDMzLCJtM3NkcCI6NSwibTNvZWVwIjo1NiwibTN0aHJwIjozNiwibTNhdmFwIjo5OSwibTNxdWFwIjo1NCwibTNzdHMiOjMsIm0zYV9zdHMiOjEsIm0zYl9zdHMiOjIsIm0zdG90YmJsbXN1bSI6MjQyMCwibTR0dCI6MTU2LCJtNHB0Ijo4OCwibTRmdCI6MzQxOSwibTRzZHAiOjMsIm00b2VlcCI6NDMsIm00dGhycCI6NjAsIm00YXZhcCI6NDIsIm00cXVhcCI6OTEsIm00c3RzIjoyLCJtNGFfc3RzIjoxLCJtNGJfc3RzIjoxLCJtNHRvdGJibG1zdW0iOjMzNzZ9"}
2021-07-27T19:29:02.651 [Information] {"dt_utc":"2021-07-27T19:29:02.2018947Z","stationID":"EdgeDevice1L","m1tt":150,"m1pt":73,"m1ft":1553,"m1sdp":16,"m1oeep":70,"m1thrp":53,"m1avap":20,"m1quap":34,"m1sts":3,"m1a_sts":2,"m1b_sts":2,"m1totbblmsum":1272,"m2tt":31,"m2pt":74,"m2ft":2520,"m2sdp":2,"m2oeep":99,"m2thrp":53,"m2avap":28,"m2quap":30,"m2sts":1,"m2a_sts":2,"m2b_sts":2,"m2totbblmsum":2760,"m3tt":121,"m3pt":49,"m3ft":3033,"m3sdp":5,"m3oeep":56,"m3thrp":36,"m3avap":99,"m3quap":54,"m3sts":3,"m3a_sts":1,"m3b_sts":2,"m3totbblmsum":2420,"m4tt":156,"m4pt":88,"m4ft":3419,"m4sdp":3,"m4oeep":43,"m4thrp":60,"m4avap":42,"m4quap":91,"m4sts":2,"m4a_sts":1,"m4b_sts":1,"m4totbblmsum":3376}
2021-07-27T19:29:02.689 [Information] Executed 'Functions.Testing1' (Succeeded, Id=734108e4-45a0-456d-b320-793b7c7768c9, Duration=39ms)
2021-07-27T19:29:03.485 [Information] Executing 'Functions.Testing1' (Reason='EventGrid trigger fired at 2021-07-27T19:29:03.4856467+00:00', Id=17dae2aa-1a74-4fbf-ad47-44adfc9ed8ce)

2021-07-27T19:29:03.486 [Information] {"properties": {"sequenceNumber": "454"},"systemProperties": {"iothub-connection-device-id": "EdgeDeviceL1","iothub-connection-module-id": "TestModule1","iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id": "637629209519885184","iothub-enqueuedtime": "2021-07-27T19:29:03.23Z","iothub-message-source": "Telemetry"},"body": "eyJkdF91dGMiOiIyMDIxLTA3LTI3VDE5OjI5OjAyLjIwMTg5NDdaIiwic3RhdGlvbklEIjoiRWRnZURldmljZTFMIiwibTF0dCI6MTUwLCJtMXB0Ijo3MywibTFmdCI6MTU1MywibTFzZHAiOjE2LCJtMW9lZXAiOjcwLCJtMXRocnAiOjUzLCJtMWF2YXAiOjIwLCJtMXF1YXAiOjM0LCJtMXN0cyI6MywibTFhX3N0cyI6MiwibTFiX3N0cyI6MiwibTF0b3RiYmxtc3VtIjoxMjcyLCJtMnR0IjozMSwibTJwdCI6NzQsIm0yZnQiOjI1MjAsIm0yc2RwIjoyLCJtMm9lZXAiOjk5LCJtMnRocnAiOjUzLCJtMmF2YXAiOjI4LCJtMnF1YXAiOjMwLCJtMnN0cyI6MSwibTJhX3N0cyI6MiwibTJiX3N0cyI6MiwibTJ0b3RiYmxtc3VtIjoyNzYwLCJtM3R0IjoxMjEsIm0zcHQiOjQ5LCJtM2Z0IjozMDMzLCJtM3NkcCI6NSwibTNvZWVwIjo1NiwibTN0aHJwIjozNiwibTNhdmFwIjo5OSwibTNxdWFwIjo1NCwibTNzdHMiOjMsIm0zYV9zdHMiOjEsIm0zYl9zdHMiOjIsIm0zdG90YmJsbXN1bSI6MjQyMCwibTR0dCI6MTU2LCJtNHB0Ijo4OCwibTRmdCI6MzQxOSwibTRzZHAiOjMsIm00b2VlcCI6NDMsIm00dGhycCI6NjAsIm00YXZhcCI6NDIsIm00cXVhcCI6OTEsIm00c3RzIjoyLCJtNGFfc3RzIjoxLCJtNGJfc3RzIjoxLCJtNHRvdGJibG1zdW0iOjMzNzZ9"}
2021-07-27T19:29:03.486 [Information] {"dt_utc":"2021-07-27T19:29:02.2018947Z","stationID":"EdgeDevice1L","m1tt":150,"m1pt":73,"m1ft":1553,"m1sdp":16,"m1oeep":70,"m1thrp":53,"m1avap":20,"m1quap":34,"m1sts":3,"m1a_sts":2,"m1b_sts":2,"m1totbblmsum":1272,"m2tt":31,"m2pt":74,"m2ft":2520,"m2sdp":2,"m2oeep":99,"m2thrp":53,"m2avap":28,"m2quap":30,"m2sts":1,"m2a_sts":2,"m2b_sts":2,"m2totbblmsum":2760,"m3tt":121,"m3pt":49,"m3ft":3033,"m3sdp":5,"m3oeep":56,"m3thrp":36,"m3avap":99,"m3quap":54,"m3sts":3,"m3a_sts":1,"m3b_sts":2,"m3totbblmsum":2420,"m4tt":156,"m4pt":88,"m4ft":3419,"m4sdp":3,"m4oeep":43,"m4thrp":60,"m4avap":42,"m4quap":91,"m4sts":2,"m4a_sts":1,"m4b_sts":1,"m4totbblmsum":3376}
2021-07-27T19:29:03.492 [Information] Executed 'Functions.Testing1' (Succeeded, Id=17dae2aa-1a74-4fbf-ad47-44adfc9ed8ce, Duration=6ms)
2021-07-27T19:29:32.499 [Information] Executing 'Functions.Testing1' (Reason='EventGrid trigger fired at 2021-07-27T19:29:32.4977236+00:00', Id=84e6b0f3-49e8-4a56-8387-f23d96b334c4)
jsquire commented 3 years ago

Hi @nainesh72. Thank you for the additional details; since this appears to be related to behavior with IoT Hub, I'll need to defer to the IoT Hub team as the experts in this area.

jsquire commented 3 years ago

//cc: @drwill-ms

nainesh72 commented 3 years ago

Sir Today also get some message duplicate if i share document whare i do

nainesh72 commented 3 years ago

image

image

nainesh72 commented 3 years ago

Hi any solution or suggestion for this problem Nainiesh

nainesh72 commented 3 years ago

Hi To day i again get duplicate record 2021-07-29T20:13:23.356 [Information] {"properties": {"iothub-creation-time-utc": "2021-07-29T20:13:22.8968104+00:00","sequenceNumber": "239"},"systemProperties": {"iothub-connection-device-id": "EdgeDeviceL1","iothub-connection-module-id": "TestModule1","iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id": "637630959233622444","iothub-enqueuedtime": "2021-07-29T20:13:23.192Z","iothub-message-source": "Telemetry"},"body": "eyJkdF91dGMiOiIyMDIxLTA3LTI5VDIwOjEzOjIyLjg5NjQ0MTlaIiwic3RhdGlvbklEIjoiRWRnZURldmljZTFMIiwibTF0dCI6ODcsIm0xcHQiOjM5LCJtMWZ0Ijo0NzMwLCJtMXNkcCI6MTQsIm0xb2VlcCI6NDksIm0xdGhycCI6MjcsIm0xYXZhcCI6NjMsIm0xcXVhcCI6NTIsIm0xc3RzIjoyLCJtMWFfc3RzIjoyLCJtMWJfc3RzIjoxLCJtMXRvdGJibG1zdW0iOjM5MTgsIm0ydHQiOjU2LCJtMnB0Ijo5MCwibTJmdCI6NDA1MiwibTJzZHAiOjIyLCJtMm9lZXAiOjYzLCJtMnRocnAiOjMzLCJtMmF2YXAiOjY0LCJtMnF1YXAiOjQ5LCJtMnN0cyI6MywibTJhX3N0cyI6MiwibTJiX3N0cyI6MiwibTJ0b3RiYmxtc3VtIjoyNzQ5LCJtM3R0IjoxMjAsIm0zcHQiOjMwLCJtM2Z0IjoyODcwLCJtM3NkcCI6MTgsIm0zb2VlcCI6NzgsIm0zdGhycCI6NDEsIm0zYXZhcCI6NzMsIm0zcXVhcCI6MiwibTNzdHMiOjEsIm0zYV9zdHMiOjEsIm0zYl9zdHMiOjIsIm0zdG90YmJsbXN1bSI6MTEyNSwibTR0dCI6NjAsIm00cHQiOjg1LCJtNGZ0IjoxMDgxLCJtNHNkcCI6MTgsIm00b2VlcCI6NjAsIm00dGhycCI6OTEsIm00YXZhcCI6OTksIm00cXVhcCI6NzYsIm00c3RzIjozLCJtNGFfc3RzIjoxLCJtNGJfc3RzIjoxLCJtNHRvdGJibG1zdW0iOjMxMDZ9"} 2021-07-29T20:13:23.357 [Information] {"dt_utc":"2021-07-29T20:13:22.8964419Z","stationID":"EdgeDevice1L","m1tt":87,"m1pt":39,"m1ft":4730,"m1sdp":14,"m1oeep":49,"m1thrp":27,"m1avap":63,"m1quap":52,"m1sts":2,"m1a_sts":2,"m1b_sts":1,"m1totbblmsum":3918,"m2tt":56,"m2pt":90,"m2ft":4052,"m2sdp":22,"m2oeep":63,"m2thrp":33,"m2avap":64,"m2quap":49,"m2sts":3,"m2a_sts":2,"m2b_sts":2,"m2totbblmsum":2749,"m3tt":120,"m3pt":30,"m3ft":2870,"m3sdp":18,"m3oeep":78,"m3thrp":41,"m3avap":73,"m3quap":2,"m3sts":1,"m3a_sts":1,"m3b_sts":2,"m3totbblmsum":1125,"m4tt":60,"m4pt":85,"m4ft":1081,"m4sdp":18,"m4oeep":60,"m4thrp":91,"m4avap":99,"m4quap":76,"m4sts":3,"m4a_sts":1,"m4b_sts":1,"m4totbblmsum":3106} 2021-07-29T20:13:23.368 [Information] Executed 'Functions.Testing1' (Succeeded, Id=d033747f-844c-40d1-8ff0-2e1cdf2e4d0e, Duration=12ms) 2021-07-29T20:14:18.878 [Information] Executing 'Functions.Testing1' (Reason='EventGrid trigger fired at 2021-07-29T20:14:18.8785579+00:00', Id=422b3574-f4f7-4134-970a-0f5e7225bb93) 2021-07-29T20:14:18.878 [Information] {"properties": {"iothub-creation-time-utc": "2021-07-29T20:13:22.8968104+00:00","sequenceNumber": "239"},"systemProperties": {"iothub-connection-device-id": "EdgeDeviceL1","iothub-connection-module-id": "TestModule1","iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id": "637630959233622444","iothub-enqueuedtime": "2021-07-29T20:14:18.562Z","iothub-message-source": "Telemetry"},"body": "eyJkdF91dGMiOiIyMDIxLTA3LTI5VDIwOjEzOjIyLjg5NjQ0MTlaIiwic3RhdGlvbklEIjoiRWRnZURldmljZTFMIiwibTF0dCI6ODcsIm0xcHQiOjM5LCJtMWZ0Ijo0NzMwLCJtMXNkcCI6MTQsIm0xb2VlcCI6NDksIm0xdGhycCI6MjcsIm0xYXZhcCI6NjMsIm0xcXVhcCI6NTIsIm0xc3RzIjoyLCJtMWFfc3RzIjoyLCJtMWJfc3RzIjoxLCJtMXRvdGJibG1zdW0iOjM5MTgsIm0ydHQiOjU2LCJtMnB0Ijo5MCwibTJmdCI6NDA1MiwibTJzZHAiOjIyLCJtMm9lZXAiOjYzLCJtMnRocnAiOjMzLCJtMmF2YXAiOjY0LCJtMnF1YXAiOjQ5LCJtMnN0cyI6MywibTJhX3N0cyI6MiwibTJiX3N0cyI6MiwibTJ0b3RiYmxtc3VtIjoyNzQ5LCJtM3R0IjoxMjAsIm0zcHQiOjMwLCJtM2Z0IjoyODcwLCJtM3NkcCI6MTgsIm0zb2VlcCI6NzgsIm0zdGhycCI6NDEsIm0zYXZhcCI6NzMsIm0zcXVhcCI6MiwibTNzdHMiOjEsIm0zYV9zdHMiOjEsIm0zYl9zdHMiOjIsIm0zdG90YmJsbXN1bSI6MTEyNSwibTR0dCI6NjAsIm00cHQiOjg1LCJtNGZ0IjoxMDgxLCJtNHNkcCI6MTgsIm00b2VlcCI6NjAsIm00dGhycCI6OTEsIm00YXZhcCI6OTksIm00cXVhcCI6NzYsIm00c3RzIjozLCJtNGFfc3RzIjoxLCJtNGJfc3RzIjoxLCJtNHRvdGJibG1zdW0iOjMxMDZ9"} 2021-07-29T20:14:18.879 [Information] {"dt_utc":"2021-07-29T20:13:22.8964419Z","stationID":"EdgeDevice1L","m1tt":87,"m1pt":39,"m1ft":4730,"m1sdp":14,"m1oeep":49,"m1thrp":27,"m1avap":63,"m1quap":52,"m1sts":2,"m1a_sts":2,"m1b_sts":1,"m1totbblmsum":3918,"m2tt":56,"m2pt":90,"m2ft":4052,"m2sdp":22,"m2oeep":63,"m2thrp":33,"m2avap":64,"m2quap":49,"m2sts":3,"m2a_sts":2,"m2b_sts":2,"m2totbblmsum":2749,"m3tt":120,"m3pt":30,"m3ft":2870,"m3sdp":18,"m3oeep":78,"m3thrp":41,"m3avap":73,"m3quap":2,"m3sts":1,"m3a_sts":1,"m3b_sts":2,"m3totbblmsum":1125,"m4tt":60,"m4pt":85,"m4ft":1081,"m4sdp":18,"m4oeep":60,"m4thrp":91,"m4avap":99,"m4quap":76,"m4sts":3,"m4a_sts":1,"m4b_sts":1,"m4totbblmsum":3106} 2021-07-29T20:14:18.889 [Information] Executed 'Functions.Testing1' (Succeeded, Id=422b3574-f4f7-4134-970a-0f5e7225bb93, Duration=11ms) 2021-07-29T20:14:19.072 [Information] Executing 'Functions.Testing1' (Reason='EventGrid trigger fired at 2021-07-29T20:14:19.0725657+00:00', Id=a898a682-9410-4499-b286-0adf2b53efd4)

nainesh72 commented 3 years ago

Hi Good Morning Any Update, Thnaks

drwill-ms commented 3 years ago

Hi - I just got back from vacation. Sorry for the late response. I'll need to pull in someone from the service team to address this question. Please wait while I find someone who can help.

kdonthineni commented 3 years ago

@nainesh72 From your sample payloads, it appears to be messages are unique since they were having different enqueued timestamps, if I understand correctly you have an event grid subscription enabled for your IoT hub with azure function as the hook and you are seeing these 2 events in your azure function. Can you please clarify what you meant by a trigger from Azure Event Hub?

nainesh72 commented 3 years ago

hi yea sir event grid subscription and from there telemetry event i execute function and that function send data to cosmos DB image image

kdonthineni commented 3 years ago

I think you meant trigger from Azure IoTHub in your initial question instead of Azure EventHub. I am highly confident that these are unique messages but if you need more details along with logs, please feel free to initiate a customer support request with the IoTHub service.

nainesh72 commented 3 years ago

Hi David R. Williamson Have an idea, when I go too deep into this problem I found that when edgeHub problem please give any view because my edge module send one message to edgeHub but how the same message send two times to the cloud because when i check that message receive DateTime(IOT Hub equetime)on cloud hub is different

drwill-ms commented 3 years ago

@nainesh72 I'm not sure I understand what you said.

Are you setting a unique MessageId property? Do you have local logging enabled? That could help you determine if a message you send is being sent twice.

nainesh72 commented 3 years ago

Hi Sir I check on the local log messages where my edge module send message to edgeHub in message routing and in the log I check that it sends only one message but edge hub send this message twice to cloude and it will be not continuously, after one or two-hour break edge hub send a duplicate message not fix time interval or regularly, so

nainesh72 commented 3 years ago

I send a Screen short of Duplicate message in my database image

nainesh72 commented 3 years ago

I already set unique message-id and unit DateTime with each message, when messages are generate

nainesh72 commented 3 years ago

Sir In my message property and message are following string dataBuffer = JsonConvert.SerializeObject(tempData); var eventMessage = new Message(Encoding.UTF8.GetBytes(dataBuffer)); eventMessage.Properties.Add("sequenceNumber", count.ToString()); eventMessage.CreationTimeUtc = DateTime.UtcNow; Console.WriteLine($"\t{DateTime.Now.ToLocalTime()}> Sending message: {count}, Body: [{dataBuffer}]"); await ioTHubModuleClient.SendEventAsync("output1", eventMessage); so i track that when and which message are duplicate send to by iothub

nainesh72 commented 3 years ago

Hi Thanks to All Now my problem is solved, it some login problem Thanks to all for your support, I some changes in my edge module send message logic then the problem solved

marceloes commented 2 years ago

Hi Thanks to All Now my problem is solved, it some login problem Thanks to all for your support, I some changes in my edge module send message logic then the problem solved

What was the solution for this issue?