Proper usage of --files-from

[wpccolorblind]

I have a list of files I would like to sync from s3 but I have too many files to use the sync command performantly.

Can I use --files-from for this? What is the usage?

I'm doing:

s3cmd sync s3://path/to/dir /path/to/dir --files-from ~/Documents/Shared\ Playground\ Data/external-datasets/ZOOM_DATA/fileupdatelist.txt

--> Where fileupdatelist.txt has the list of relative file paths.

However, I can't seem to figure out any method of calling s3cmd that respects this parameter, which seems like the only thing I actually need.

If I pass --debug to the above it doesnt look like it even checks the parameter.

If I change the sync to a get it also doesnt look like it does anything with parameter and I get:

"ERROR: Parameter problem: Expecting S3 URI with a filename or --recursive: s3://zoomcloud/ZOOM_DATA/zoomuploads/"

Thanks!

[fviard]

"Read list of source-file names from FILE. Use - to read from stdin." The help is probably not clear, but checking the source code, --files-from is only implemented for "local" sources.

It is less convenient, but maybe in your case you can have usage of the --exclude, --include flags?

[wpccolorblind]

I wasn't able to get this to work performantly, but I can check again. Say I have a list of 50,000 files I want to sync from a server that has 100s of directories each of which have many directories inside of them going 3-5 layers deep. I know the files I want I just want to get them.

Is it expected that a combination of include/exclude flags would work performantly? When I did some tests even with a few files it seemed to need to perform way too many checks.

I was trying exclude "*" and then include of specific file names.

Thanks for the assistance and digging into the --files-from stuff.