Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.25k stars 4.59k forks source link

Possible Memory leak when downloading blobs #45204

Open JannikBreuer opened 1 month ago

JannikBreuer commented 1 month ago

Library name and version

Azure.Storage.Blobs 12.21.1

Describe the bug

Our application needs to download and upload many images from a blob storage. We used the following logic for this: (We have also tried the stream variant, but the same problem occurs)


    private async Task CopyBlobStorageFilesFromSourceToMlStudioBlobStorageAsync(List<LabelProcess> labelsToExport)
    {
        this._logger.LogInformation("Start uploading images to ml blob storage");

        foreach (var labelToExport in labelsToExport)
        {
            this._logger.LogInformation("Uploading image {ImageFileName} to ml blob storage", labelToExport.RootContainer.Image.FileName);

            var content = await this._azureBlobStorageImageSourceContainerProxy.GetBlobClient(labelToExport.RootContainer.Image.FileName).DownloadContentAsync();

            // Todo: Do something with the content

            this._logger.LogInformation("Finished uploading image {ImageFileName} to ml blob storage", labelToExport.RootContainer.Image.FileName);
        }
    }

I have left out the upload part, as the download part is already causing the problem.

Locally, I have executed this method several times and left the API running for a long time afterwards. I would have assumed that after a short time it would release/delete the byte arrays it needed to download the images. But unfortunately, it doesn't, which is why the memory consumption keeps increasing:

This is the memory usage after starting my API:

image

This is the memory usage after executing the method mentioned above a couple of times:

image

As you can see in the picture, the garbage collector clears up the heap, but the unmanaged memory remains the same size.

Am I missing something or am I doing something wrong?

Expected behavior

Memory is cleaned up by the garbage collection

Actual behavior

Memory is not released/cleared again

Reproduction Steps

Loading many images multiple times.

Environment

.Net 8 - ASP .NET API

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

JannikBreuer commented 1 month ago

I also created a minimal API to make sure that the problem is not on our side, but the problem is still there:

app.MapGet("/blobStorageTest", async () =>
    {
        var _azureBlobStorageImageSourceContainerProxy = new BlobContainerClient("ConnectionString", "container");

         var arrayWithManyImages = new List<string>
        {
            "TestImage1.jpg",
        };

        foreach (var image in arrayWithManyImages)
        {
            Response<BlobDownloadResult> content = await _azureBlobStorageImageSourceContainerProxy.GetBlobClient(image).DownloadContentAsync();
        }

    })
    .WithName("blobstorageTest")
    .WithOpenApi();

image