search

skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1.09k stars 62 forks source link

Azure transfer verificaion #493

Closed parasj closed 2 years ago

parasj commented 2 years ago

I suspect this issue is caused by far too high parallelism for object store queries. This may be fixed as we switched to RBAC authentication.

Tracked by SKY-224

 _____ _   ____   _______ _       ___   _   _  _____
/  ___| | / /\ \ / / ___ \ |     / _ \ | \ | ||  ___|
\ `--.| |/ /  \ V /| |_/ / |    / /_\ \|  \| || |__
 `--. \    \   \ / |  __/| |    |  _  || . ` ||  __|
/\__/ / |\  \  | | | |   | |____| | | || |\  || |___
\____/\_| \_/  \_/ \_|   \_____/\_| |_/\_| \_/\____/

Will transfer 1152 objects totaling 69.76GB from azure:eastus to azure:westus
    VMs to provision: 1x azure:eastus, 1x azure:westus
    Estimated egress cost: $1.40 at $0.02/GB
    fake_imagenet/train-00000-of-01024 => train-00000-of-01024
    fake_imagenet/train-00001-of-01024 => train-00001-of-01024
    fake_imagenet/train-00002-of-01024 => train-00002-of-01024
    fake_imagenet/train-00003-of-01024 => train-00003-of-01024
    ...
    fake_imagenet/validation-00124-of-00128 => validation-00124-of-00128
    fake_imagenet/validation-00125-of-00128 => validation-00125-of-00128
    fake_imagenet/validation-00126-of-00128 => validation-00126-of-00128
    fake_imagenet/validation-00127-of-00128 => validation-00127-of-00128

Using overridden docker image: ghcr.io/skyplane-project/skyplane:local-7ab4bd091242ee5f5832a558525190ee
Instances will remain up and may result in continued cloud billing. Remember to call `skyplane deprovision` to deprovision gateways.
Storing debug information for transfer in /tmp/skyplane/transfer_logs/20220628_011227/client.log
✓ Installing gateway package (2/2) in 12.34s
✓ Assigning gateways permissions to buckets (2/2) in 4.55s
🚀 69.76GB transfer job launched
Warning: It can take up to 60s for role assignments to propagate on Azure. See issue #355
  Transfer progress (1152 of 1152 chunks) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 69.8/69.8 GiB 1.6 GB/s 0:00:00
Compression saved 0.47% of egress fees
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
...
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
✓ Verifying transfer (1152/1152) in 186.15s
abiswal2001 commented 2 years ago

fixed in #505 and #507