'Transfer-Encoding: chunked' support

[trentm]

Should be able to have large file uploads stream via a pipe without knowing the length/size.

See mark's email "ANN: Streaming uploads" to manta-private-beta.

[tgross]

I'll be looking into this as part of my work on triton-mysql

[misterbisson]

@trentm is there anything from "mark's email" that should be moved into this ticket?

[trentm]

the body of that email (which had been sent to the early beta users of Manta):

This morning we pushed an updated manta storage stack that allows for
"streaming" uploads (meaning HTTP transfer-encoding: chunked).  The
semantics of Manta with regards to PUT requests are now such that if
you send a Content-Length header, then nothing has changed.  If you
send a Transfer-Encoding: chunked header, then you do not need to know
the length of the object ahead of time, as Manta will keep track of it
for you, and record the real size at the end.  One caveat: by default,
the maximum length of an object when using streaming uploads is 5Gb;
this can be overridden with a new header 'max-content-length', which
you can use to "reserve" enough space for your object.

The node-manta repo (https://github.com/joyent/node-manta)  is updated
to support this, so please git pull/reinstall it to leverage this. The
CLI works with it as well, as an example (from the root of
node-manta):

$ cat README.md | ./bin/mput -H 'content-type: text/plain'
/mcavage/stor/README.md

Will work just fine.  Note that "content-type" can't be determined in
this case since node-manta just gets a byte stream, so whereas before
you were able to leverage MIME type lookups, you'll almost assuredly
want to explicitly set the content-type now.