Output stream closed when stream passed to aws s3

[EivydasV]

Version information

• fluent-ffmpeg version: 2.1.2
• ffmpeg version: The lastest
• OS: Linux

Code to reproduce

import { Upload } from '@aws-sdk/lib-storage';
import { PassThrough } from 'node:stream';
import ffmpeg from 'fluent-ffmpeg';

    const format = 'webm';
    const command = ffmpeg()
      .input(fs.createReadStream(data.filePath))
      .format(format)
      .outputOptions([
        '-vf scale=1280x720',
        '-b:v 1024k',
        '-minrate 512k',
        '-maxrate 1485k',
        '-tile-columns 2',
        '-g 240',
        '-threads 8',
        '-quality good',
        '-crf 32',
        '-c:v libvpx-vp9',
        '-c:a libopus',
        '-speed 4',
      ])
      .pipe()
      .on('progress', (progress) => {
        console.log(progress);
      })
      .on('end', function () {
        console.log('Video conversion complete');
      })
      .on('error', function (err) {
        console.log('An error occurred: ' + err.message);
      });

    const folder = path.join(AwsFolder.VIDEOS, data.userId, data.videoId);
    const fileName = `${nanoid()}_${data.scale}.${format}`;
    const fullPath = path.join(folder, fileName);
    const upload = new Upload({
      client: this.s3BucketService.client,
      params: {
        Bucket: 'youtube-nest',
        Key: fullPath,
        Body: command as PassThrough,
      },
    });

    await upload.done();

When I run this I am getting Error: Output stream closed(from fluent-ffmpeg), error occurs after end event is emitted. Looks like that Passthrough closes stream before is finished (I am using @aws-sdk/client-s3)

[EivydasV]

I can see that there is timeout in node_modules setTimeout(function() { emitEnd(new Error('Output stream closed')); ffmpegProc.kill(); }, 20); });. I changed timeout to 1000 and now it works, however this solution is not ideal at all, because you need to edit node_modules, which is no no

[mariyan-borisov]

Ever since I've switched the Docker image from mhart/alpine-node:10 to node:16-alpine (that probably affects the ffmpeg version as well), this issue is also present here. Seeing the activity in this project, I've worked around this issue using a temporary file - downloading the file, feeding it to ffmpeg, and deleting it. I also got to the same code as you did, but didn't manage to find a proper fix for it. Adding more delay seems like an ugly workaround.

[EivydasV]

@mariyan-borisov I did the same workaround as you. First upload file to local storage (without using stream as output) and when processing is finished then I start upload to aws s3. I do not like this workaround either it should work with streams, but it does not. Library maintainer also do not care, library last time was updated 5 years ago. I very doubt it that he will fix this issue now. This is basically dead library. Unfortunately I did'n manage to find any replacement for this library :(

[tlvince]

I ran into this as well with AWS SDK v3 (AWS SDK v2 does not seem to be affected).

As a workaround, I'm calling the ffmpeg binary directly (as fluent-ffmpeg appears to be unmaintained), which works for simple cases e.g.:

const { spawn } = require('child_process')
const { Upload } = require('@aws-sdk/lib-storage')

const ffmpeg = spawn('ffmpeg', ['-i', 'input.mp4', 'pipe:1'])
const upload = new Upload({
  params: {
    Body: ffmpeg.stdout
  }
})

[ed0wolf]

I had some luck with telling fluent-ffmepg not to close the output stream and then manually calling .end() on the output stream when the ffmpeg command is ended. Something like:

const outputStream = new PassThrough();

ffmpeg(input)
  .noVideo()
  .outputFormat('wav')
  .output(outputStream, { end: false })
  .on('end', () => {
    outputStream.end();
  })
  .run();

const upload = new Upload({
  params: {
    Body: outputStream
  }
})

[ricfio]

The terminateTimeout option (PR https://github.com/fluent-ffmpeg/node-fluent-ffmpeg/pull/1292) could solve this issue.