search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.67k stars 237 forks source link

What is the size of the listing queue for batch operations? #5

Closed baragona closed 6 years ago

baragona commented 6 years ago

A series of LIST commands needs to be sent to S3 to get the list of files to GET...

Then that is usually put into a queue and workers handle them.

So I'm wondering, how big is the queue in this application and can it be changed?

seruman commented 6 years ago

WorkerPool has unbuffered channel for jobs that workers to get.

https://github.com/peakgames/s5cmd/blob/f907aea99ae85bd27e62c389f71c9e2ffd8b5d58/core/worker.go#L33-L37

So, if you want to manage number concurrent jobs running, you can change the numWorkers parameter of WorkerPool via numworkers command line flag. (defaults to 256) https://github.com/peakgames/s5cmd/blob/f907aea99ae85bd27e62c389f71c9e2ffd8b5d58/s5cmd.go#L171-L179

https://github.com/peakgames/s5cmd/blob/f907aea99ae85bd27e62c389f71c9e2ffd8b5d58/s5cmd.go#L61