FFMPEG pipe stream to S3 pre-signed PUT URL.

[xoraingroup]

Version information

• fluent-ffmpeg version: 2.1.2
• ffmpeg version: 4.1.3
• OS: Mac OS X 10.14.5

Code to reproduce

const ffmpeg = require('fluent-ffmpeg');
const request = require('request');

const video = '/Users/xorain/Documents/test_videos/movie.mp4';
const putURL = 'https://some-aws-signed-put-url/videos-bucket-nurm/219/219-1564216171442-6dfd8c309a25fa9779fa7164acd35221f1c4301d3a037e297456637c236ee9da.mp4?X-Amz-Meta-Uid=219&X-Amz-Meta-Sub=614505702013204&X-Amz-Meta-Vid=2604&X-Amz-Meta-Ofn=movie.mp4&X-Amz-Meta-Region=eu-nurm-de&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20190727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190727T082931Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=79243c973634d0fee5d54c0f6c99e7fb657e425553e2281bf0aa4d01186b49ea';

const outputPipe = ffmpeg(video, {logger: console})
    .audioCodec('copy')
    .videoCodec('copy')
    .outputOptions([
         '-movflags frag_keyframe+empty_moov',
         '-movflags +faststart'
     ])
    .toFormat('mp4')
    .on('start', (cmd) => {
        console.info(cmd)
    })
    .on('error', (error) => {
        console.info(error)
    })
    .on('end', () => {
        console.info('finished uploading')
    })
    .pipe(request.put(putURL))

(note: if the problem only happens with some inputs, include a link to such an input file)

Expected results

The stream should be piped to amazon AWS put url.

Observed results

Error: You cannot pipe to this stream after the outbound request has started. at Request. (/usr/local/lib/node_modules/request/request.js:495:26) at Request.emit (events.js:198:13) at Socket.Readable.pipe (_stream_readable.js:742:8) at processCB (/Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/processor.js:480:31) at /Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/processor.js:213:7 at FfmpegCommand.proto._getFfmpegPath (/Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/capabilities.js:90:14) at FfmpegCommand.proto._spawnFfmpeg (/Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/processor.js:132:10) at /Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/processor.js:437:12 at /Users/xorain/Git/personal/ffmpeg-logging/node_modules/async/dist/async.js:473:16 at next (/Users/xorain/Git/personal/ffmpeg-logging/node_modules/async/dist/async.js:5329:29) Emitted 'error' event at: at errorOrDestroy (internal/streams/destroy.js:107:12) at Request.onerror (_stream_readable.js:717:7) at Request.emit (events.js:198:13) at Request. (/usr/local/lib/node_modules/request/request.js:495:12) at Request.emit (events.js:198:13) [... lines matching original stack trace ...] at FfmpegCommand.proto._spawnFfmpeg (/Users/xorain/Git/personal/ffmpeg-logging/node_modules/fluent-ffmpeg/lib/processor.js:132:10)

Checklist

• [X] I have read the FAQ
• [X] I tried the same with command line ffmpeg and it works correctly (hint: if the problem also happens this way, this is an ffmpeg problem and you're not reporting it to the right place)
• [x] I have included full stderr/stdout output from ffmpeg

I would like to know if its possible to directly pipe the output stream to a signed put url?? Just curious to know. Thank you

[Mat-thieu]

Hi I could be mistaken, but I feel like request.put() does not actually return a stream you can write to. You have to make sure the request becomes a stream input and the request library knows it should return an input.

I did a quick google search but couldn't find a quick solution, maybe you will have more luck. Let me know if it worked!

[Rambarani]

@Mat-thieu Did you got the solution ?

[anand-panigrahi]

@xoraingroup hey, were you able to solve this ?

[rabeetcarma]

any hope on the solution?

[gulshan-dev123789]

@Mat-thieu Did you got the solution ?

any solution how to sream output to bucket ?

[Mat-thieu]

@gulshan-dev123789

I don't have a solution, I only remember request doesn't provide what you need here. It won't give you a writable stream.

Instead you can consider using the https package in node, which should return you a writable stream according to the docs https://nodejs.org/api/https.html#httpsrequestoptions-callback

probably looks something like this, but I didn't test it fully

const https = require('https');
const { URL } = require('url');

const putURL = new URL('https://upload-url');

const options = {
    hostname: putURL.hostname,
    path: putURL.pathname + putURL.search,
    method: 'PUT',
    headers: {
        'Content-Type': 'video/mp4',
    },
};

console.log(options);

// creates a writable stream
const req = https.request(options, (res) => {
    res.on('data', (chunk) => {
        console.log(`BODY: ${chunk}`);
    });
    res.on('end', () => {
        console.log('No more data in response.');
    });
});

req.on('error', (e) => {
    console.error(`error`, e);
});

then you can pipe the request to ffmpeg

ffmpeg(video)
   // your operations
    .on('error', (error) => {
        console.error(`ffmpeg error: ${error}`);
        req.end();
    })
    .on('end', () => {
        console.info('ffmpeg processing finished, finishing upload...');
        req.end();
    })
    .pipe(req, { end: false });