MemoryError in ADO build agent when downloading 4+ GB SQL bacpac file from storage account using az storage fs file download

[tbrothers]

Describe the bug

The az storage fs file download command worked fine a few months ago when I was downloading a 1.9GB sql bacpac file from our storage account to the build machine, but lately the bacpac file has grown to over 4 GB and I'm getting a memory error when trying to download the file to the agent machine.

Related command

az storage fs file download -p database.bacpac -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login

Errors

ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream MemoryError To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Issue script & Debug output

az storage fs file download -p database.bacpac -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login --debug

DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream MemoryError

ERROR: cli.azure.cli.core.azclierror: The command failed with an unexpected error. Here is the traceback: ERROR: az_command_data_logger: The command failed with an unexpected error. Here is the traceback: ERROR: cli.azure.cli.core.azclierror: Traceback (most recent call last):

Expected behavior

The az storage fs download command should download the bacpac file to the agent build machine.

After that is completely, the import to SQL database command is run using SqlPackage.exe Action Import...

Environment Summary

azure-cli 2.57.0

core 2.57.0 telemetry 1.1.0

Extensions: azure-devops 0.26.0

Dependencies: msal 1.26.0 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @tbrothers, wondering how much free memory is left for the machine/vm at the time of downloading? Would the az storage blob download command work for your use case? It might be a bit faster as well. Still trying to replicate this MemoryError.

[arrwhidev]

I am experiencing the same issue as @tbrothers. I can see the python process memory usage spiking before the MemoryError happens. It appears that az storage fs file download is attempting to download the file into memory entirely before writing it to disk. This is definitely going to cause issues with large files (>1GB in my case).

I just switched over to az storage blob download as suggested by @calvinhzy and this is working perfectly.