search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

Faster Delete for Piped Filenames #75

Closed william-silversmith closed 2 years ago

william-silversmith commented 2 years ago

I was hoping cloudfiles would be faster than gsutil, but they're about the same for this operation:

cat files.txt | cloudfiles rm - --progress
cat files.txt | gsutil -m rm -I

This is because cloudfiles is deleting them one at a time. There should be a way to group simple cases and mass delete them.

william-silversmith commented 2 years ago

Got this working. 0ec776657f3149b787df54324cbc35610284ae80