After the initial data set release we determined that while SFTP can work it is limited and less performant than other options available. In order to make the process faster we will provide users with the option to download the dataset using the Azure native file transfer tool azcopy. Based on our findings this will reduce the amount of time it takes to download the data by approximately 25%. This also opens the option to allow users to transfer the data to other storage medium, like their our Azure storage containers or even AWS S3 bucket.
Acceptance Criteria:
When the dataset is ready for download we will provide the user with a SAS URI that they can use with the command line tool azcopy
The URI will be set to expire after three days.
An example command will be provided to users
A link to the azcopy tool will be provided along with a link to the azcopy documentation.
After the initial data set release we determined that while SFTP can work it is limited and less performant than other options available. In order to make the process faster we will provide users with the option to download the dataset using the Azure native file transfer tool
azcopy. Based on our findings this will reduce the amount of time it takes to download the data by approximately 25%. This also opens the option to allow users to transfer the data to other storage medium, like their our Azure storage containers or even AWS S3 bucket.Acceptance Criteria:
azcopy