search

TritonDataCenter / node-manta-sync

Rsync style command for Joyent's Manta
31 stars 9 forks source link

remote info gathering is unnecessarily slow #7

Open bahamas10 opened 10 years ago

bahamas10 commented 10 years ago

info gathering (step 2) can be sped up immensely by doing client.ftw() once at the beginning and caching the results, as opposed to doing a HEAD for every local file found.

bahamas10 commented 9 years ago

or, as pointed out in #manta on freenode, don't HEAD files that can't possible exist... for example if you have local files like

foo/
foo/a
foo/b
foo/c

check to see if foo/ exists on the remote end as a directory... if it doesn't don't check any of the subfiles because they can't possibly exist.