aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.54k stars 4.13k forks source link

aws s3 sync status takes time to calculate total/remaining files #7957

Open yoni-yad2 opened 1 year ago

yoni-yad2 commented 1 year ago

Describe the bug

Running aws s3 sync [s3 folder with 8 files] [local folder] and for approx half of the runtime the status doesn't show real numbers of total/remaining files

Expected Behavior

Status shown at the start: Completed xxx GiB/~yyy GiB (zzz MiB/s) with ~8 file(s) remaining (8 files)

Current Behavior

While it copies the first 5 files in parallel the status shown is: Completed xxx GiB/~yyy GiB (zzz MiB/s) with ~5 file(s) remaining (calculating...)

Reproduction Steps

Upload 10 big files (10 GB each) to an s3 folder and run from and ec2 VM: aws s3 sync [s3 folder] [local folder]

Possible Solution

aws s3 sync should try first scanning for a few seconds all files/subfolders and calculate total number of files and size to quickly display it in the command status.

If scanning takes more than a few seconds then it should start syncing the files/subfolders and display with total (calculating...) file(s) remaining (calculating...) and eventually display the real total/remaining number after if finished scanning the entire folder tree.

Additional Information/Context

No response

CLI version used

2.11.25

Environment details (OS name and version, etc.)

Windows Server 2022 Datacenter / Powershell

vaibhav0077 commented 1 year ago

Hi @hssyoo, Can I work on this task ?