IoTHub/EventHub should provide ability to add offset as parameter

[brandonh-msft]

Although the trigger allows us to get the offset of the incoming message, we cannot set the offset of the desired message. This makes testing messages either from the IoTHub or Function side challenging, it also means that reprocessing messages because of a bad function or skipping messages because earlier messages do not need to be processed (or the data from IoTHub is bad) is not possible. Further, there may be a scenario when an IoT device comes online and I want to skip the first X number of messages because only the most recent one is the one I actually need to process. Adding this parameter helps control cost, compute and test scenarios.

Ex.

I create a new function and I have a full day of messages. I know that the first 1,000 messages will not work with this function so I want to skip over them without using compute/being charged for the fnc processing those 1,000. In fact, it may throw an error on the first 1,000.

Second, although a new consumer group may allow me to begin at the start the beginning, I should not need to always create new consumer groups (ex. I may want to always use $Default) for this scenario.

Create an Azure Function with the EventHub or IoTHub trigger and attempt to choose a specific EventHub offset.​

Problem Statement: The partner knows that the first set of messages coming into a function from IoT Hub/ Event Hub should be ignored, so they want to be able to skip those messages. However, Azure Functions does not allow partners to skip messages in this way, and will charge the partner regardless.

Desired Experience: Partners should be able to set an offset for messages coming from Event Hub/IoT Hub.

Internal Tracking

202740

[brandonh-msft]

cc @theangellmethod

[alrod]

Can you use offset filtering inside function code and skip the messages processing by conditional clause? Offset can be stored in apps setting and you can get offset using string offset or EventData.Offset

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[angellmethod]

I won't be able to test this in code until 5/10 at the earliest. If I'm understanding correctly, it seems like if I am using a conditional clause to skip messages, then I would still get charged for execution since the function still has to execute X number of times before it gets to the data I actually want to use. If I have to keep doing this for testing, it would eat into that allocation rather quickly. Because of that, it seems better to be able to choose the offset without having to get charged for the function call. If my understanding is incorrect, please let me know. I won't be able to test the recommended method until 5/10 at the earliest.

[nzthiago]

This might be related to https://github.com/Azure/azure-webjobs-sdk/issues/1240

[brandonh-msft]

ah yeah certainly seems like a dupe... but extensions/bindings should have their issues created here, no? 🤷‍♂️

[drdamour]

probably dupe of https://github.com/Azure/azure-functions-eventhubs-extension/issues/64

[alrod]

Fixed in https://github.com/Azure/azure-functions-eventhubs-extension/issues/64