extract frames with pipe in Android

[kingfisherphuoc]

Description I am trying to use pipe to extract frames from a video to get better-extracting performance. It took me 1s to extract a 1080x720 frame from the video. So, my idea is to use pipe protocol and while extracting with ffmpeg command, I have another thread to read the pipe as bitmap. Is it possible?

This is what I did in Android:

String pipe = Config.registerNewFFmpegPipe(context);
FFmpeg.execute("ffmpeg -re -y -i rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov -f image2pipe -r 1 " + pipe);

And then, I read the pipe in another thread:

if (fileInputStream == null) {
      fileInputStream = new BufferedInputStream(new FileInputStream(pipe)); // open pipe once
}
String value = convertStreamToString(fileInputStream); // read from pipe

However, this code new FileInputStream(pipe) took forever to finish. Is there something wrong with my code?

Environment

• Platform: Android
• Version: 4.4.LTS

[kingfisherphuoc]

@tanersener do you have any suggestions?

[tanersener]

I don't know what the problem is. Have you tried using a smaller video file, does that make a change?

Additionally, what do you do on the thread that runs the ffmpeg command. Do you stop it at some point or does it continue running?

[kingfisherphuoc]

@tanersener The BickBuckBunny video is small (200x400) I would say, but it took really long to open the inpustream. I did run the command in a new thread ( it's RxJava -Schedulers.io() ). Actually, it run perfectly till the end of the video. While the ffmpeg command was running, I created a new thread to read InputStream from the pipe. However, the new created thread stuck at opening the pipe. The thread was continued only after the ffmpeg command finished. The problem here is that I cannot open and read the output pipe file during ffmpeg command is running.

new thread() --> runs ffmpeg

another thread() --> new BufferedInputStream(new FileInputStream(pipe)); // this only be passed if the ffmpeg command finished. 

I can open the pipe file only if ffmpeg command ended. Does the library lock the pipe file until the ffmpeg command finished?

[tanersener]

I can open the pipe file only if ffmpeg command ended. Does the library lock the pipe file until the ffmpeg command finished?

Unfortunately, this is the behaviour of ffmpeg. It also happens when you read from a pipe. I explained this on the Pipe Support wiki page.

Please note that ffmpeg will block the execution of your command in step #3 until some data is available in your pipe. So if you don't have a step #4, thread executing step #3 will wait indefinitely.

[kingfisherphuoc]

@tanersener as far as I understand, you meant that there's no way to read the pipe file while the ffmpeg is running. Is it right? In this case, do you suggest any command to output the result into in-memory object such as bytes? Because I know the memory I can use, there's not a memory issue. If we can output the result directly into memory without writing it to a file, we can overcome the file read/write speed limited. This is a great library. I would love to use it instead of aother library.

[tanersener]

First of all, there is no way to get the result as a Java object right now.

And, regarding the pipes. I don't know what your problem is there. Because if you're using a file in your ffmpeg command, it will eventually finish, doesn't it? Unless you're planning to use a stream instead of a file.

[kingfisherphuoc]

@tanersener Sorry, I might make you confused. What I tried to archive was extracting all frames from a video (rtsp or http) and while extracting frames, I would like display every frame into an ImageView (it's like showing a sequence of frames in an imageview). The current issue is that it took near a second to extract and save a frame to a file in sdcard and it caused the imageview's play laggy (1 fps) --> I would like to improve it by using pipe. However, the problem is that while extracting frames with pipe, I could not read the pipe. So, can I read the pipe content while executing ffmpeg command or there might be a better idea?

[tanersener]

I don't know. I haven't tried doing that.

[kingfisherphuoc]

@tanersener do you intent to add any kind of jni function to support Java object from ffmpeg command in the future?

[tanersener]

Options I have right now are not promising. If I find a good technical solution I'll implement it.

[alexcohn]

@kingfisherphuoc if it takes 1 sec to save each frame on the sdcard, no pipes and no JNI will help you. This time is far beyond the expected latency of sdcard write and read.

You can verify this easily: write a binary file of garbage of the expected size of a frame to the same partition, and read it back. I expect you will see few milliseconds for the whole operation, though this depends on your hardware.

Most likely, your bottleneck is in rtsp: access.

Update: for me, this video on https://www.wowza.com/html/mobile.html is stuck.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.