Optimizing render time: Piping frames to ffmpeg

[polarby]

Piping images to FFmpeg instead of saving each file to source and then feeding it to FFmpeg. See here for more info.

FFmpegKit currently lacks documentation for piping input. Here is the current approach to pipe input, but it will only show the first piped image in the mp4.

final Directory directory = await getApplicationDocumentsDirectory();
    final pipe = await FFmpegKitConfig.registerNewFFmpegPipe();

    Future<void> pipeFiles() async {
      final imageFile0 = await getImageFileFromAssets("image0.jpeg");
      await FFmpegKitConfig.writeToPipe(imageFile0.path, pipe!);
      final imageFile1 = await getImageFileFromAssets("image1.jpeg");
      await FFmpegKitConfig.writeToPipe(imageFile1.path, pipe);
    }

    final process =  FFmpegKit.executeAsync('-y -framerate 1 -f image2pipe -i $pipe -frames 2 ${directory.path}/out11.mp4',
        (session) async {
      await FFmpegKitConfig.closeFFmpegPipe(pipe!);
    });

    await pipeFiles();

    final file = File('/data/user/0/com.example.example/app_flutter/out11.mp4');

[tienthanh1993]

hi @polarby, Thank you for your great work! I have found a way to pipe frames to FFmpeg. I think that calling FFmpegKitConfig.writeToPipe includes an EOF signal which causes FFmpeg to stop receiving new images.

I hope this helps!

//getFileBuffer read file buffer from assets
Future<Uint8List> getFileBuffer(String assetName) async {
    final ByteData data = await rootBundle.load(assetName);
    return data.buffer.asUint8List();
}

final Directory directory = await getApplicationDocumentsDirectory();
final pipe = await FFmpegKitConfig.registerNewFFmpegPipe();

 Future<void> pipeFiles() async {
   final fifo = File(pipe!); // open pipe file from ffmpeg
   final stream = fifo.openWrite();
   final imageFile0 = await getFileBuffer("image0.jpeg");
   stream.add(imageFile0 );
   final imageFile1 = await getFileBuffer("image1.jpeg");
   stream.add(imageFile1); 
   // ... 
   await stream.close(); // close the stream will make ffmpeg stop 
}

final process =  FFmpegKit.executeAsync('-y -framerate 1 -f image2pipe -i $pipe -frames 2 ${directory.path}/out11.mp4',
        (session) async {
      await FFmpegKitConfig.closeFFmpegPipe(pipe!);
});

await pipeFiles();

final file = File('/data/user/0/com.example.example/app_flutter/out11.mp4');

[polarby]

Thanks for the input. I am very tied up with work rn. And although the integration might exponentially decrease the rendering time, and quality (as frames no longer need to be stored), this might require a bit of work to integrate (RenderProcessor and RenderCapturere merge). I could though imagine that with integration and drastically improving this plugin, thinking of this plugin become THE go-to plugin when it comes to capturing video/images as widgets. Feel free to implement this, I will make sure to merge it timely/give you the permissions to do so!

[polarby]

@tienthanh1993 In case you are currently working on this. The issue I have mentioned might require this feature in order to support the web!

[tienthanh1993]

@polarby I am making a prototype, but it is not as fast as the one reported in your Readme.md. This may be due to differences in the MotionSettings.

[polarby]

I am not sure what speed you are referring to "in my Readme.md"? So does this mean piping takes longer? This seem very unlikely to me, are you sure?

[tienthanh1993]

I am not sure what speed you are referring to "in my Readme.md"? So does this mean piping takes longer? This seem very unlikely to me, are you sure?

this one

[polarby]

This might be device and settings related though. I would instead compare the rendering time on your device piping/non-piping.

Piping should remove temporary writing, reading frames, and loss of RAM of saved frames (also time-related)?! I see no reason for a longer process time...

[tienthanh1993]

hi @polarby,

I got it working and sent a pull request. I hope you get some ideas.

[polarby]

Closing this in favor of #15. Piping is possible, but may require special consideration of the end user - and is essential for web usage