s3 sync --delete not deleting empty folders

[jardakotesovec]

Follow up to #2533

We are using aws-cli on OS X to sync bucket to local folder. So if there is file within folder, it gets synced properly. Issue is when file gets deleted - empty folder locally remains, which overtime can result in many empty folders.

aws s3 sync --delete s3://bucket/folder .

[stealthycoin]

Marking as a feature request.

[robertd]

+1

[jeffbildz]

+1

[cyuste]

+1

[paulmaunders]

+1

[paulmaunders]

I'm also experiencing this running the AWS command on CentOS.

We have an S3 bucket which contains daily db backups. We are using S3 lifecycle management on the bucket to expire each daily backup folder after 7 days.

We then have a CentOS server on GCP which is periodically syncing the bucket using the following command...

aws s3 sync s3://bucket-name /backups/s3/bucket-name --delete

The individual db backup files are being deleted, but the daily folders remain...

# du -h --max-depth=1
0       ./2017-10-10_2300
0       ./2017-10-11_2300
0       ./2017-10-12_2300
0       ./2017-10-13_2300
0       ./2017-10-14_2300
0       ./2017-10-15_2300
0       ./2017-10-16_2300
0       ./2017-10-17_2300
0       ./2017-10-18_2300
0       ./2017-10-19_2300
0       ./2017-10-20_2300
0       ./2017-10-21_2300
0       ./2017-10-22_2300
0       ./2017-10-23_2300
0       ./2017-10-24_2300
0       ./2017-10-25_2300
5.6G    ./2017-10-26_2300
5.6G    ./2017-10-27_2300
5.6G    ./2017-10-28_2300
5.6G    ./2017-10-29_2300
5.6G    ./2017-10-30_2300
5.6G    ./2017-10-31_2300
5.5G    ./2017-11-01_2300
39G     .

These folders no longer exist on S3....

[root@backups ~]# aws s3 ls bucket-name/mysql-backups/
                           PRE 2017-10-26_2300/
                           PRE 2017-10-27_2300/
                           PRE 2017-10-28_2300/
                           PRE 2017-10-29_2300/
                           PRE 2017-10-30_2300/
                           PRE 2017-10-31_2300/
                           PRE 2017-11-01_2300/

I would expect sync --delete to delete the folders too.

[CameronGo]

+1

[cbarensfeld]

+1

[rrybalkin]

+1

[jusaho]

+1

[ntman4real]

+1

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168367-s3-sync-delete-not-deleting-empty-folders

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[walljcg]

+1

[pmspire]

+1

[jkeller-miovision]

+1

[mikebrickwall]

+1

[thugbeatz]

+1

[myheartsgoon]

+1

[serhiireva]

+1

[kellenarb]

+1 And as a quick note to anyone else looking for a quick workaround, the following will delete any empty folders as a cleanup step: find . -type d -empty -delete

[trimeloni]

+1

[gladwynb]

Definitely +1

[Feldhacker]

The reverse it true as well: aws s3 sync --delete . s3://bucket/folder When syncing from a local filesystem to S3, any folders that were deleted locally are NOT deleted from S3 as we would expect.

[gentaro-sakamoto]

+1

[hhcordero]

I hope this feature request will be considered.

aws s3 sync --delete . s3://bucket

For the command above, it does not make sense to leave empty "folders" in s3 bucket if we're going to follow what the definition of aws sync means. Looking at the description of aws sync cli it says:

Syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.

I think the inconsistency lies on what does this line really means "Syncs directories and S3 prefixes.". To me I would expect the folder in S3 to be deleted if no file in that folder exist on the local directory.

[kecsap]

It is quite ridiculous that this basic feature has not been implemented yet until this day. Especially for paying customers.

[jolleye]

It is quite ridiculous that this basic feature has not been implemented yet until this day. Especially for paying customers.

I couldn't agree more!

[vitorbezerra]

+1

[adamnellis]

I am encountering this bug as well. This is not a feature request. It is clearly a bug.

My situation:

1. Delete local directory: dist/test/
2. Run aws s3 sync dist s3://bucket-name/ --acl public-read --delete

Expected result:

• Directory test should be deleted from the bucket

Observed result:

• Directory test still exists in the bucket as an empty directory. Any files that it previously contained have been deleted.

[florianspecker]

+1

[ghost]

+1

[cmccoypdx]

+1

[snarkeystarkey]

FWIW, same happens with aws s3 mv /local/path/ s3://uri/

/local/path will contain empty directories upon completion.

[waffel]

+1

[ioMatrix]

Please do something about this. Having to script deletion of empty folders is silly IMO.

[michaelgiraldo]

+1

[ezorita]

+1

[josiekre]

+1

[ghost]

+1

[frantzmiccoli]

+1

[tomekit]

+1

[michalz92]

+1

[Cabe]

+1

Still an issue 4 years on? Really?

[CatiaFerreira6]

+1

[igrishaev]

I'm another one who has faced the issue. At first glance, syncing with S3 looked promising, but those empty directories turned my file collection into a mess as I often move files. Indeed, I cannot believe this issue has not been fixed yet, especially when every single user of this utility pays money to AWS.

[ifs-aminebouarfa]

+1 It is really ridiculous to see that the demand is pending since 2017...

[leads]

+1 to remove empty folders with sync --delete please

[paul-uz]

+1

[schonz]

+1