Closed wbira closed 7 months ago
Hi Waldemar,
Thanks for reporting this issue. Just wanted to give you a quick update on our investigation.
The exception is from Azure Storage, originating from this call
var resource = await rg.GetStorageAccountAsync(asset.Data.StorageAccountName, cancellationToken: cancellationToken);
return GetStorageAccount(resource);
Apparently, there is a limit on the amount of storage account management operations that one can do with storage (https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling), in particular, we think that you hit the 800 requests / 5 min throttling limit.
We don't believe there is any way to raise these limits.
Because the storage operation is used pretty pervasively in the code, it is a bit difficult to fully address this issue in the short run. We'll try to modify the code (when possible) to cache the calls to storage management api and reuse but even so, we're not sure we can avoid hitting this throttling limit fully. (unless we do retry and sleep for 5 minutes)
In the interim, there are two mitigations,
there is this -b flag,
-b, --batch-size <batch-size> Batch size for parallel processing. [default: 5]
that you can try to set to 1
This should slow down the number of asset analyzed / migrated because it will do it '1' asset at a time.
The other mitigation is to use
-cs, --creation-time-start
to limit the number of assets per run, that way you can chop the number of assets by date. This way you can partition the asset you want to migrate by date range and you run the tool multiple times, each with a smaller date range.
We apologize for your inconvenience..
Hi Waldemar,
We committed a PR to help alleviate the issue (https://github.com/Azure/azure-media-migration/pull/239). Feel free to try it out to see if it helps with your issue.
Hello @pohhsu Today I've managed to create next run of analyze command on staging media services account and it went through
2024-02-22 12:28:07.878 +00:00 [DBG] Analyzing Assets: 31894/0 Assets
2024-02-22 12:28:07.888 +00:00 [DBG] Finished analysis of assets for account: xxxxxx. Time taken "00:19:21.9585590"
I have a feeling that your change in PR #239 also improved performance a lot! Tomorrow I will run full migration to check if also assets command is also working (if it will be successul I will close a ticket), but it seems that issue is resolved (at least for analyze) Thank you for quick response and support!
Everything works fine. Thank you @pohhsu for your help!
Hello! I run analyze command using migration tool against our staging media service account. It contains around 30k assets there. After 12k I got following exception. I'm a bit worried, cause on production media services account we have around 140k assets. Is there any way to increase limits (if yes which limit should be raised), or handle this error more gracefully e.g by implementing exponetial backoff or some kind of batching? I run this job on Azure container Instance with standard cpu/memory settings (4cpus 16GB RAM)