Support multiple queue triggers with coordination

[victorhurdugaci]

Feature proposal:

Today, it is very easy to write to multiple queues from a webjob. That allows me to have a workflow where 1 webjob generates work for more than 1 job.

What's missing is the opposite: multiple jobs generate work for a single job.

Basically, I would like to trigger a function when two or more messages are available in two different queues and these messages meet some condition.

Here is an example:

Queues: Q1 and Q2
Message: M11 (in Q1), M12 (in Q2), M21 (in Q1), M22 (in Q2)
Webjob function: F that listens on both Q1 and Q2.

F can get triggered by any of the following combinations of messages:

M11 and M12
M21 and M22

But not

M11 and M22
M21 and M11

So, I don't want F to be triggered by any two random messages in Q1 and Q2 but only by messages in the form MX1 and MX2 (where X is the same digit).

For my particular case, I have a field on the two messages that is identical and that correlates messages from different queues.

My 2c syntax proposal:

public void F([QueueTrigger("correlationId", "Q1", "Q2") Payload p) { ... }

where

QueueTrigger(string correlationField, params string[] queues)

I think it is very tricky to implement because you would need a way to guarantee that the same job host gets both messages from a queue. Also, the two messages might not be at the top of the queue so you would have to delay the processing for a while (forever if a message is lost?) until you have all the triggers

[mathewc]

This sounds like another Service Bus Session scenario. There was another issue logged here recently related to this: https://github.com/Azure/azure-webjobs-sdk/issues/529. Using Service Bus sessions, you can have multiple messages grouped in the same "session" (where you control the session ID), and you can wait until all messages in the session are received before beginning processing. If we updated the ServiceBusTrigger to support this, would this address your scenario?

[isaacabraham]

A slight variation on this would be to have a collection of messages on the same queue that when completed generate another queue message.