Issue with Raw Image Streaming to FFmpeg Process via Pipe on React Native

[uripont]

Title: Challenges with Raw Image Streaming to FFmpeg in React Native Application

Description:

Goal

We aim to implement a solution for capturing and encoding video directly within a React Native application. The objective is to stream raw image data directly into an FFmpeg process for continuous video encoding, ensuring minimal storage overhead and efficient processing.

Problem

1. Pipe Auto Closes with PNGs: When attempting to stream PNG images to the FFmpeg process, the pipe auto-closes after each image is passed. This behavior disrupts the continuous encoding process, resulting in incomplete video streams.

2. Inability to Pass Base64 Raw Images: In our current implementation we attempted to pass as base64 encoded raw images. However, the FFmpeg process expects a file path rather than raw data when passing it with FFmpegKitConfig.writeToPipe(), making it impossible to feed the images directly into the encoder. This limits our ability to keep the encoding process active until explicitly closed.

The ideal solution involves continuously streaming raw image data directly to FFmpeg without the need for intermediate storage. This would involve:

• Configuring FFmpeg to accept raw video input via a pipe.
• Ensuring the pipe remains open for continuous data feed until manually closed.
• The encoder keeps taking in the raw inputs and using hardware-accelerated libraries to encode them into a video.

Challenges

• FFmpeg Configuration: Properly configuring FFmpeg to handle raw image data streams, including setting appropriate pixel formats and frame dimensions, is causing issues.
• Data Feed: Developing a mechanism to pass raw image data (in base64 or other formats) directly into FFmpeg without requiring file paths (and thus without requiring them to be stored on the user's file system).

We seek guidance on:

• Configuring FFmpeg to handle continuous raw image data streams.
• Efficiently passing base64 raw images or similar data formats directly to FFmpeg.
• Ensuring the encoding process remains active until manually closed, without auto-closing the pipe.

Environment:

• React Native
• ffmpeg-kit-react-native

Expected Behavior: Continuous streaming of raw image data to FFmpeg, allowing seamless video encoding until the process is manually terminated.

Actual Behavior: Pipes auto-close after each PNG frame, and raw image data cannot be directly passed due to file path requirements.

Any advice or solutions to address these issues would be greatly appreciated. Thank you!

[Indra-poudel]

@uripont did you able to come with solution. I have similar usecase :D

[uripont]

Unfortunately no, we haven't found any solution to this issue.

But we have found an alternative way to fulfill our needs: instead of encoding a video ourselves frame by frame, we end up re-encoding an already recorded video to achieve the same result.