Storage Queue Batch Binder

[Bio2hazard]

Hey,

the lack of a batch binder for storage queues is problematic for us when it comes to doing some vendor integrations, so I wanted to ask if there are any plans to add that functionality in the near future.

As an example, our email provider gives us a rate limit of 200 calls per minute, but a single call can contain a payload of up to 200 email addresses in an array. The provider responds by returning a Key-Value-Pair of failed sends, with the key being the position in the submitted array, and the value being the error message.

I saw that a few months back support was added to receive blobs as a IEnumerable, and it would be wonderful if something similar could be done for storage queues.

To clarify, what I need is not a "wait until we have X messages before processing them", but rather a "process up to X messages if they are available".

I have built this functionality out in a Worker Role before, but would prefer to leverage the web jobs SDK moving forward.

If I can, I would like to contribute on making this happen.

What do you think?

[mathewc]

Note that for the IEnumerable Blob bindings, those are only supported for the non-trigger input/output bindings. None of the input trigger bindings trigger on batches. However, a community member recently wrote a batching Queue trigger (see https://github.com/ealsur/WebJobs.Extensions.GroupQueueTrigger) using the extensibility model. You might give that a try.

If there is enough interest I do think this is something that would make sense to be added as part of the core triggers.

[Bio2hazard]

I hadn't seen the Batching Queue Trigger, but yeah, that's basically the functionality I was looking for.

I will work with the extension for now, and of course would be thrilled to see the functionality make it's way into the core.

Thank you!

[agnauck]

I am also looking for such a feature. Taking a look at the batching Queue trigger linked above now.

Thanks, Alex

[SeanFeldman]

👍 for this binder to be added.

Scenario I'm in - periodic work that could be requested multiple times, but shouldn't be executed multiple times. All requests are collected as messages and received in a bacth. Let's say webjob received 20 messages, only 1 work item will be executed, saving 19 other ones from been executed.

[koverda]

I too am looking for a feature such as this.

[BrianVallelunga]

I'd love something like this for Azure Service Bus queues as well.

[tezizzm]

I too am looking for a feature such as this.

[alexsorokoletov]

+1 for this feature, glad to see it is available as an extension

[mikhailshilkov]

Would be useful for Azure Functions too, and custom triggers aren't supported there

[jeremy001181]

Any updates please? Is it still sitting in the backlog?

[snympi]

Any update please?

[czb182]

I agree this would be useful. I'm dealing with millions of messages potentially, and to use storage queue trigger, I would need to batch them up, otherwise I would overwhelm other services processing one at a time at the throughput level i need.

[hibearpanda]

This would be an important feature for us as well.

[onionhammer]

This would be extremely valuable, definitely a big thumbs up!

[nagarjunacspeaks]

Looks like this is pending since long time. Though the one with service bus batch trigger got implemented. https://github.com/Azure/azure-functions-servicebus-extension/issues/15 Any idea if this is going to implemented in near future?

[loopjockey]

:+1: this would be great

[adriennn]

go vote https://feedback.azure.com/forums/34192--general-feedback/suggestions/42573244-storage-queue-batch-storage-queue-batch-binder

[SeanFeldman]

@adriennn, it would be better to consolidate the effort and have upvotes on the issue instead of monitoring two different sources. I would suggest doing the opposite and directing from user feedback site here.

[snympi]

@adriennn @SeanFeldman I voted at both locations.

[adriennn]

@SeanFeldman thanks for the headsup, it's always difficult to judge which of the feedback platforms are used to take decisions.

btw, for these developping with old sdk, there's QueueBatch https://github.com/Scooletz/QueueBatch

[Herman-Wu]

This feature is very useful in our project too.

[SamambaMan]

Would like this feature as well, specially for Python worker

[shishirpy]

Need this feature.