search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.47k stars 4.8k forks source link

[BUG] Can't bulk delete a blob with space in it's name using BlobItem.Name #22329

Closed poddubitskya closed 3 years ago

poddubitskya commented 3 years ago

Describe the bug When bulk deleting blobs operation using BlobItem.Name fails with an error if one of the blobs has space in it's name

A workaround is wrapping BlobItem.Name with WebUtility.UrlEncode

Same code worked before with these NuGet packages used: Azure.Storage.Blobs.Batch 12.5.1 Microsoft.Azure.Storage.Blob 11.2.2

Expected behavior Should be able to delete any blob existing using it's name that was previously returned by the SDK (BlobItem.Name)

Actual behavior (include Exception or Stack Trace)

Azure.RequestFailedException: One of the request inputs is not valid.
RequestId:[GUID]
Time:[TIME]
Status: 400 (One of the request inputs is not valid.)
ErrorCode: InvalidInput

Content:
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidInput</Code><Message>One of the request inputs is not valid.
RequestId:[GUID]
Time:[TIME]</Message></Error>

Headers:
x-ms-error-code: InvalidInput
x-ms-request-id: [GUID]
x-ms-version: 2020-08-04
x-ms-client-request-id: [GUID]
Content-Length: 221
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0

   at Azure.Storage.Blobs.Specialized.BlobBatchClient.UpdateOperationResponses(IList`1 messages, Response rawResponse, Stream responseContent, String responseContentType, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchInternal(BlobBatch batch, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchAsync(BlobBatch batch, Boolean throwOnAnyFailure, CancellationToken cancellationToken)
   at TransPerfect.Media.Dubbing.Infra.Azure.Blobs.AzureBlockBlobsFileRepository.DeleteFolderWithContentAsync(String path) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Infra.Azure\Blobs\AzureBlockBlobsFileRepository.cs:line 166
   at TransPerfect.Media.Dubbing.Dal.Core.ProjectContentFilesRepository.DeleteAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Dal\Core\ProjectContentFilesRepository.cs:line 91
   at TransPerfect.Media.Dubbing.Business.Core.ProjectContentManager.DeleteAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Business\Core\ProjectContentManager.cs:line 66
   at TransPerfect.Media.Dubbing.Services.Core.ProjectService.DeleteProjectAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Services\Core\ProjectService.cs:line 244
   at TransPerfect.Media.Dubbing.API.Controllers.Core.ProjectsController.DeleteAsync(Guid projectId) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.API\Controllers\Core\ProjectsController.cs:line 166
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)

To Reproduce

  1. Upload a block blob with a space in it's name
  2. Read the blob data into a BlobItem using an instance of BlobContainerClient
  3. Try performing a batch delete using BlobItem.Name as a parameter for the BlobBatch.DeleteBlob method
      // _containerClient is a BlobContainerClient instance
      // ListBlobsAsync uses: _containerClient.GetBlobsAsync(BlobTraits.None, BlobStates.None, path).AsPages(default, 100);
      List<BlobItem> allBlobs = await ListBlobsAsync(path, "*.*");
      try
      {
        BlobItem[][] blobBatches = SplitIntoBatches(allBlobs, MaxRequestsPerBatch);

        foreach (BlobItem[] blobBatch in blobBatches)
        {
          var batch = new BlobBatch(_blobBatchClient);
          foreach (var blobItem in blobBatch)
          {
            batch.DeleteBlob(_containerClient.Name, blobItem.Name); // Blob name with space goes here
          }
          await _blobBatchClient.SubmitBatchAsync(batch); //Exception
        }
      }

Environment:

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
**Describe the bug** When bulk deleting blobs operation using BlobItem.Name fails with an error if one of the blobs has space in it's name A workaround is wrapping BlobItem.Name with WebUtility.UrlEncode ***Same code worked before with these NuGet packages used:*** Azure.Storage.Blobs.Batch 12.5.1 Microsoft.Azure.Storage.Blob 11.2.2 **Expected behavior** Should be able to delete any blob existing using it's name that was previously returned by the SDK (BlobItem.Name) **Actual behavior (include Exception or Stack Trace)** ```csharp Azure.RequestFailedException: One of the request inputs is not valid. RequestId:[GUID] Time:[TIME] Status: 400 (One of the request inputs is not valid.) ErrorCode: InvalidInput Content: InvalidInputOne of the request inputs is not valid. RequestId:[GUID] Time:[TIME] Headers: x-ms-error-code: InvalidInput x-ms-request-id: [GUID] x-ms-version: 2020-08-04 x-ms-client-request-id: [GUID] Content-Length: 221 Content-Type: application/xml Server: Windows-Azure-Blob/1.0 at Azure.Storage.Blobs.Specialized.BlobBatchClient.UpdateOperationResponses(IList`1 messages, Response rawResponse, Stream responseContent, String responseContentType, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken) at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchInternal(BlobBatch batch, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken) at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchAsync(BlobBatch batch, Boolean throwOnAnyFailure, CancellationToken cancellationToken) at TransPerfect.Media.Dubbing.Infra.Azure.Blobs.AzureBlockBlobsFileRepository.DeleteFolderWithContentAsync(String path) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Infra.Azure\Blobs\AzureBlockBlobsFileRepository.cs:line 166 at TransPerfect.Media.Dubbing.Dal.Core.ProjectContentFilesRepository.DeleteAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Dal\Core\ProjectContentFilesRepository.cs:line 91 at TransPerfect.Media.Dubbing.Business.Core.ProjectContentManager.DeleteAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Business\Core\ProjectContentManager.cs:line 66 at TransPerfect.Media.Dubbing.Services.Core.ProjectService.DeleteProjectAsync(Guid projectID) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.Services\Core\ProjectService.cs:line 244 at TransPerfect.Media.Dubbing.API.Controllers.Core.ProjectsController.DeleteAsync(Guid projectId) in C:\_dev\_sources\Media.DubbingProject\src\TransPerfect.Media.Dubbing.API\Controllers\Core\ProjectsController.cs:line 166 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context) at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) ``` **To Reproduce** 1. Upload a block blob with a space in it's name 2. Read the blob data into a BlobItem using an instance of BlobContainerClient 3. Try performing a batch delete using BlobItem.Name as a parameter for the BlobBatch.DeleteBlob method ```csharp // _containerClient is a BlobContainerClient instance // ListBlobsAsync uses: _containerClient.GetBlobsAsync(BlobTraits.None, BlobStates.None, path).AsPages(default, 100); List allBlobs = await ListBlobsAsync(path, "*.*"); try { BlobItem[][] blobBatches = SplitIntoBatches(allBlobs, MaxRequestsPerBatch); foreach (BlobItem[] blobBatch in blobBatches) { var batch = new BlobBatch(_blobBatchClient); foreach (var blobItem in blobBatch) { batch.DeleteBlob(_containerClient.Name, blobItem.Name); // Blob name with space goes here } await _blobBatchClient.SubmitBatchAsync(batch); //Exception } } ``` **Environment:** - Name and version of the Library package used: Azure.Storage.Blobs.Batch 12.6.0 - Hosting platform or OS and .NET runtime version: .NET SDK 5.0.202 (db7cc87d51), Windows 10.0.19042, Host 5.0.5 (2f740adc14) - IDE and version : JetBrains Rider 2021.1.3 Build #RD-211.7442.29
Author: poddubitskya
Assignees: -
Labels: `Client`, `Service Attention`, `Storage`, `customer-reported`, `needs-team-attention`, `needs-triage`, `question`
Milestone: -
amishra-dev commented 3 years ago

Hi @poddubitskya, thanks for reporting this. you are right that it should work and url encoding the name is a good workaround.
The change for fixing this is a behavior change. We will talk about it internally and get back to you here. @kasobol-msft fyi

poddubitskya commented 3 years ago

Hi @poddubitskya, thanks for reporting this. you are right that it should work and url encoding the name is a good workaround. The change for fixing this is a behavior change. We will talk about it internally and get back to you here. @kasobol-msft fyi

Kudos!

Thank you