polarby / render

A package to render any native static and moving flutter widgets to exportable formats
https://pub.dev/packages/render
MIT License
50 stars 28 forks source link

Optimizing render time: Piping frames to ffmpeg #3

Closed polarby closed 1 year ago

polarby commented 1 year ago

Piping images to FFmpeg instead of saving each file to source and then feeding it to FFmpeg. See here for more info.

FFmpegKit currently lacks documentation for piping input. Here is the current approach to pipe input, but it will only show the first piped image in the mp4.

final Directory directory = await getApplicationDocumentsDirectory();
    final pipe = await FFmpegKitConfig.registerNewFFmpegPipe();

    Future<void> pipeFiles() async {
      final imageFile0 = await getImageFileFromAssets("image0.jpeg");
      await FFmpegKitConfig.writeToPipe(imageFile0.path, pipe!);
      final imageFile1 = await getImageFileFromAssets("image1.jpeg");
      await FFmpegKitConfig.writeToPipe(imageFile1.path, pipe);
    }

    final process =  FFmpegKit.executeAsync('-y -framerate 1 -f image2pipe -i $pipe -frames 2 ${directory.path}/out11.mp4',
        (session) async {
      await FFmpegKitConfig.closeFFmpegPipe(pipe!);
    });

    await pipeFiles();

    final file = File('/data/user/0/com.example.example/app_flutter/out11.mp4');
tienthanh1993 commented 1 year ago

hi @polarby, Thank you for your great work! I have found a way to pipe frames to FFmpeg. I think that calling FFmpegKitConfig.writeToPipe includes an EOF signal which causes FFmpeg to stop receiving new images.

I hope this helps!

//getFileBuffer read file buffer from assets
Future<Uint8List> getFileBuffer(String assetName) async {
    final ByteData data = await rootBundle.load(assetName);
    return data.buffer.asUint8List();
}
final Directory directory = await getApplicationDocumentsDirectory();
final pipe = await FFmpegKitConfig.registerNewFFmpegPipe();

 Future<void> pipeFiles() async {
   final fifo = File(pipe!); // open pipe file from ffmpeg
   final stream = fifo.openWrite();
   final imageFile0 = await getFileBuffer("image0.jpeg");
   stream.add(imageFile0 );
   final imageFile1 = await getFileBuffer("image1.jpeg");
   stream.add(imageFile1); 
   // ... 
   await stream.close(); // close the stream will make ffmpeg stop 
}

final process =  FFmpegKit.executeAsync('-y -framerate 1 -f image2pipe -i $pipe -frames 2 ${directory.path}/out11.mp4',
        (session) async {
      await FFmpegKitConfig.closeFFmpegPipe(pipe!);
});

await pipeFiles();

final file = File('/data/user/0/com.example.example/app_flutter/out11.mp4');
polarby commented 1 year ago

Thanks for the input. I am very tied up with work rn. And although the integration might exponentially decrease the rendering time, and quality (as frames no longer need to be stored), this might require a bit of work to integrate (RenderProcessor and RenderCapturere merge). I could though imagine that with integration and drastically improving this plugin, thinking of this plugin become THE go-to plugin when it comes to capturing video/images as widgets. Feel free to implement this, I will make sure to merge it timely/give you the permissions to do so!

polarby commented 1 year ago

@tienthanh1993 In case you are currently working on this. The issue I have mentioned might require this feature in order to support the web!

tienthanh1993 commented 1 year ago

@polarby I am making a prototype, but it is not as fast as the one reported in your Readme.md. This may be due to differences in the MotionSettings.

polarby commented 1 year ago

I am not sure what speed you are referring to "in my Readme.md"? So does this mean piping takes longer? This seem very unlikely to me, are you sure?

tienthanh1993 commented 1 year ago

I am not sure what speed you are referring to "in my Readme.md"? So does this mean piping takes longer? This seem very unlikely to me, are you sure?

this one

polarby commented 1 year ago

This might be device and settings related though. I would instead compare the rendering time on your device piping/non-piping.

Piping should remove temporary writing, reading frames, and loss of RAM of saved frames (also time-related)?! I see no reason for a longer process time...

tienthanh1993 commented 1 year ago

hi @polarby,

I got it working and sent a pull request. I hope you get some ideas.

polarby commented 1 year ago

Closing this in favor of #15. Piping is possible, but may require special consideration of the end user - and is essential for web usage