Scooletz / QueueBatch

WebJobs/Azure Functions trigger providing batches of Azure Storage Queues messages directly to your function
Apache License 2.0
41 stars 4 forks source link

Clarify usage #20

Open ErikAndreas opened 5 years ago

ErikAndreas commented 5 years ago

I struggled a bit to figure out how to actually get a hold of my queued message. Maybe clarify in docs (not sure if what I got is the best way...) what to do with a Memory<byte> object? Maybe my lack of c# knowledge but I think a clarification might be useful?!

so I 'd add a message as usual:

 await batchQueue.AddMessageAsync(new CloudQueueMessage(JsonConvert.SerializeObject(myDTO)));

and then consume it in my trigger

public static async Task RunAsync([QueueTrigger("batch-queue")]MyDTO myDTO, ILogger log)
{
        // use myDTO here ...
}

using BatchQueueTrigger

public static async System.Task RunAsync([QueueBatchTrigger("batch-queue")]IMessageBatch batch,  ILogger log)
{
        foreach (var msg in batch.Messages)
        {
                MyDTO dto = JsonConvert.DeserializeObject<MyDTO>(System.Text.Encoding.Default.GetString(msg.Payload.ToArray()))
        }
        batch.MarkAllAsProcessed();
}

is there a better way?

Scooletz commented 5 years ago

The IMessageBatch provides only bytes by design as QueueBatch is meant to be used with deserialization that should be able to work with data in-situ, without need of allocating or trying to get a string out of it. Preferable way to use it is to use it with some binary serialization/deserialization.

In other words, this is not a generic batch for Azure Storage Queues but rather specific, for formats that can deal with binary data. I hope this clarifies the purpose of this project @EricAndreas. If yes, feel free to close the issue.

ErikAndreas commented 5 years ago

That's great, I was not aware of that purpose (maybe state that and provide a usage example? I know it would have helped me...) I found this project when looking for batching options (rather than scale out) with queue triggers and this seemed like a perfect fit - if it also has better performance it's even better! Would you say QueueBatch.Tests/IntegrationTests.cs shows "formats that can deal with binary data"? I got usage from that but not quite sure what (other?) usage this project aims for (any samples)?

But (more importantly), great work!

Scooletz commented 5 years ago

If your serialization supports reading in zero-copy mode, or something similar, you could allocate an instance once, and use it for all the cases. Or if every message in a batch represents an array, you could read it one after another. Thinking about it as loop (batch) over loop (array) of items. These would be scenarios. The project was tested with the last approach (batches of arrays of items) but this case is unfortunately not public 😢.

If you agree, let's keep this issue open @ErikAndreas. I don't promise to address this within a week, but it'll be at least visible.