search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
607 stars 219 forks source link

AzCopy keep handling some files after those are transferred #1897

Open ms-ishiyam opened 2 years ago

ms-ishiyam commented 2 years ago

Which version of the AzCopy was used?

10.16 (Latest)

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Windows 10 (Client), Azure Files Standard (File server)

What command did you run?

Azcopy copy

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

.\azcopy.exe copy 'C:\Users\testuser11\Documents\test30M-20000*' "https://xxx.file.core.windows.net/test13/test17?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&" --recursive=true --preserve-smb-permissions=true --preserve-smb-info=true --log-level=ERROR --check-length=false --cap-mbps=300

What problem was encountered?

When I transferred 20,000 files for 30MB to Azure Files by AzCopy, I found that AzCopy process keep handling some files after those are transferred. I confirmed that the resource lock was finished after AzCopy transferred all of 20,000 files. I think this behavior is very inconvenient because we can not open those files during transfer. If the number of files is so many and the volume is big, the period of the release lock will be too long. I also confirmed that robocopy is different behavior. It releases the lock immediately after transferred but its performance is worse than AzCopy. I hope AzCopy will be enhanced this behavior.

How can we reproduce the problem in the simplest way?

You make 20,000 files for 30MB using fsutil and transfer them to Azure Files. During transfer, you try to overwrite one of the files. It can reproduce. You can find the file locked by AzCopy.exe using resource monitor.

image

Have you found a mitigation/solution?

Wait for the finish or use robocopy.

adreed-msft commented 2 years ago

Feature request: drop file handles as transfers complete

briggsb9 commented 2 years ago

I'm also seeing this behaviour in a current project. Very inconvenient during a large-scale data migration.

cdiskan commented 1 year ago

This issue still appears to be present. This is a major headache for large syncs that should be entirely invisible to end users.

MRllling commented 1 year ago

yep, i'm also seeing this phenomenon appears on linux servers, which will also decrease current jobs when running a new process

rbaret commented 12 months ago

I ran into this issue 2 weeks ago, it's still not solved.

MRllling commented 12 months ago

I ran into this issue 2 weeks ago, it's still not solved.

a feasible though is like using "nohup" command on linux or kill the process once completed...

artembrening-nc commented 9 months ago

Hello, does anybody have a solution for this issue? We are currently as well facing this issue in a large migration project.

ms-ishiyam commented 9 months ago

In my opinion, we should use another command like robocopy. We can't use AzCopy for such a large migration on Azure Files.

cdiskan commented 8 months ago

@ms-ishiyam

Performance of Robocopy (Or any SMB-based utility) is vastly inferior to azcopy's API access when migrating data across connections with latency. The higher the latency, the worse the SMB performance, and the more critical it is to have a functional way to copy data via API.