TritonDataCenter / node-manta

Node.js SDK for Manta
75 stars 54 forks source link

local file to upload('-f' option) can be stdin ('-') #324

Open cinsk opened 7 years ago

cinsk commented 7 years ago

This small commit enables to use - for -f option value so that muntar can read from standard input instead from reading from real file. In short, users can able to use following one-liner to upload files in a directory to manta.

    $ tar -C SOME-DIRECTORY -cf - . | muntar -f - ~~/MANTA-DIRECTORY

Without this commit, users should use one of following to upload a directory:

    $ tar -cf ARCHIVE.tar SOME-DIRECTORY
    $ muntar -f ARCHIVE.tar ~~/public

Or, if the shell is bash or zsh, users can use process substitution feature like this:

    $ muntar -f <(tar -cf - SOME-DIRECTORY) ~~/public
davepacheco commented 7 years ago

Thanks for the PR! FYI, this repository does not use pull requests. See CONTRIBUTING.md.

As for the change: it doesn't seem like this will actually work. We end up creating a bunch of workers, each of which reads the input file. I think the code relies on the fact that each worker will end up reading the entire tar file. If you use process.stdin (or any non-regular file), whatever one worker reads, others will miss, and I'd expect the result would be each worker reading totally invalid tar records. Is that not what happens?

cinsk commented 7 years ago

Thanks for the quick response, davepacheco@.

Strangely, it was fine to use my patch in one of my mac, but in another mac it failed. I guess that there was only 1 worker when it was successful.

Still, there is an option to control the number of workers, so I'll working on to force 1 worker if standard input is used, and follow the process you suggested.

Thank you