`pipe storage cp` shall start data upload before traversing full source hierarchy - Githubissues

epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform

https://cloud-pipeline.com

Apache License 2.0

145 stars 58 forks source link

`pipe storage cp` shall start data upload before traversing full source hierarchy #2574

Open sidoruka opened 2 years ago

sidoruka commented 2 years ago

Background At the moment, pipe storage cp/mv CLI command for --recursive operations runs (at least) in two phases:

Scan the source directory recursively and collect all the files/folders
Transfer the data collected

For huge filesystem hierarchies the scan process may take a lot of time (hours) and consume a lot of memory.

Approach It would be great to change the scanning procedure in a more asynchronous fashion, e.g.:

Change the scanning process to collect files/folders in batches (e.g. 1000 paths)
Once we have a batch collected - start upload to the destination
In the meantime (while upload is in progress) collect the next batch of files
And so on...

sidoruka commented 2 years ago

Backport to release/0.16