Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.25k stars 4.59k forks source link

[QUERY] How to read messages from AzureStorageQueue faster? #45694

Open eddynaka opened 2 weeks ago

eddynaka commented 2 weeks ago

Library name and version

Azure.Storage.Queues.12.20.0

Query/Question

Hello,

I have a solution running on AzureFunctions, .NET 8, Windows with at least 8 cores. The AzureFunction as a cron function that reads data from an azure storage queue so it can batch for further processing.

To read the messages, I'm doing something like:

while ((queue.GetProperties()).Value.ApproximateMessagesCount > 0)
{
    var messages = queue.ReceiveMessages(32);
    foreach (var message in messages.Value)
    {
        string fingerprint = message.Body.ToString();
        items.Add(fingerprint);
        queue.DeleteMessage(message.MessageId, message.PopReceipt);
    }

    // If we have been processing for more than 15 minutes, stop processing so we won't lose all the work.
    if (watch.Elapsed.TotalMinutes > maxMinutesToProcess)
    {
        break;
    }
}

What I observed is that it is taking a very long time to dequeue items from the queue. More or less 2k every 5min.

Already reviewed this: https://learn.microsoft.com/en-us/azure/storage/queues/storage-performance-checklist to see what other configuration changes I could apply for .NET 8, but most are related to .NET Framework.

So, I'm looking for help to understand how can I dequeue faster or if there is a link showing some best practices on that.

Thanks!

Environment

.NET 8 AppService/AzureFunction windows 8 cores

github-actions[bot] commented 2 weeks ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.