search

Azure / azure-storage-net-data-movement

Azure Storage Data Movement Library for .Net
MIT License
275 stars 133 forks source link

Include "Retry-After" in the Response Header when Graph Throttling occurs while moving the files from Sharepoint. #313

Open KP-Indian opened 11 months ago

KP-Indian commented 11 months ago

Which service(blob, file) does this issue concern?

In case the data movement is triggered from Sharepoint to Blob and the graph call returns throttling exception with status code as 429, graph sends the Response header "retry-after".

https://learn.microsoft.com/en-us/graph/throttling#best-practices-to-handle-throttling

The same header is not passed as the Response Header of data movement library to retry the data movement call after some time.

Which version of the SDK was used?

2.0.4

On which platform were you using? (.Net Framework version or .Net Core version, and OS version)

.Net core

How can the problem be reproduced? It'd be better if the code caused the problem can be shared.

This issue can be reproduced only by throwing throttling exception from Graph call.

What problem was encountered?

Not receiving "retry-after" header as part of response.

Have you found a mitigation/solution?

No

anthonywhite commented 2 weeks ago

We are also having this issue (SDK 2.0.5) moving files from SharePoint (using a Graph Download URL) to Azure Blob. The error we see in App Insights is

Copy status: Failed; Description: 429 TooManyRequests "Copy failed when reading the source.".
   at Microsoft.Azure.Storage.DataMovement.TransferControllers.AsyncCopyController.HandleFetchCopyStateResultAsync(StorageCopyState copyState, Boolean gotDestinationAttributes)
   at Microsoft.Azure.Storage.DataMovement.TransferControllers.AsyncCopyController.GetCopyStateAsync()
   at Microsoft.Azure.Storage.DataMovement.TransferControllers.AsyncCopyController.DoWorkInternalAsync()
   at Microsoft.Azure.Storage.DataMovement.TransferControllers.TransferControllerBase.DoWorkAsync()
   at Microsoft.Azure.Storage.DataMovement.TransferScheduler.ExecuteJobInternalAsync(TransferJob job, CancellationToken cancellationToken)
   at Microsoft.Azure.Storage.DataMovement.SingleObjectTransfer.ExecuteAsync(TransferScheduler scheduler, CancellationToken cancellationToken)
   at Microsoft.Azure.Storage.DataMovement.TransferManager.DoTransfer(Transfer transfer, TransferContext transferContext, CancellationToken cancellationToken)

Is there any way to properly handle/retry this error?