-
Hi I am trying to download lot of files from S3 to local.
I am sending the command file like below
**s5cmd run --numworker 256 commands.txt**
and my commands file will have below sample of around …
-
-
We should survey options for data transfer between tmp buckets and long-er term storage.
- fsspec
- object-store
- [skyplane](https://github.com/skyplane-project/skyplane) 🚧 @jbusecke see if we …
-
For those who are not developers, it may be helpful to have this link instead of the s5cmd github repo pointer, which we have right now: https://github.com/peak/s5cmd#installation.
-
Identified by Vamsi, Where NLST collection is having the disabled S5cmd box like below. Currently NLST is the only collection where this happens
![image](https://github.com/ImagingDataCommons/IDC-…
-
The download instructions do not match what I contributed in this PR:
https://github.com/ImagingDataCommons/IDC-WebApp/pull/1399/files
Specifically, my change had this:
and the download s…
-
There is another s3 upload/download tool called s5cmd
https://github.com/peak/s5cmd
I benchmarked uploading:
https://golang.org/dl/go1.15.5.linux-amd64.tar.gz
to the very same location in a wasabi…
-
Trying to sync a lot of files and getting
[7258676.421271] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-…
-
is there any benefits vs s5cmd? At least, s5cmd (https://github.com/peak/s5cmd) should be mentioned in the README as a possible alternative, IMO.
-
# Summary
Implement a client that supports parallelism such as [s5cmd](https://github.com/peak/s5cmd) to enable much faster S3 artifacts I/O speeds, which is crucial when working with many and\or l…