putStream with transfer-encoding chunked

ccorcos commented 7 years ago

I'm trying to stream a request response straight into S3, but I'm running into issues because the response is chunked and doesnt have a content-length:

headers:
   { 'content-type': 'image/jpeg',
     expires: 'Fri, 02 Jun 2017 22:05:16 GMT',
     date: 'Fri, 02 Jun 2017 22:05:16 GMT',
     'cache-control': 'private, max-age=0, must-revalidate, no-transform',
     vary: 'Accept, X-GData-Authorization, GData-Version',
     'gdata-version': '1.0',
     'transfer-encoding': 'chunked',
     'x-content-type-options': 'nosniff',
     'x-frame-options': 'SAMEORIGIN',
     'x-xss-protection': '1; mode=block',
     server: 'GSE',
     'alt-svc': 'quic=":443"; ma=2592000; v="38,37,36,35"',
     connection: 'close' },

Here's what I've got going (using the request package):

request.get(url)
    .on("response", (res) => {
        const path = randomID()
        const req = client.putStream(res, path, res.headers, function (err, res) {
            if (err) {
                reject(err)
            }
            else {
                resolve(req.url)
            }
        })
    })

I see you talked about it here and added that null check https://github.com/Automattic/knox/issues/92 but I got it working simply by removing that check...

@TooTallNate @rauchg

devpascoe commented 7 years ago

hmmm facing this too

devpascoe commented 7 years ago

i ended up having to save locally then putFile :(

Automattic / knox

putStream with transfer-encoding chunked #322