How to grab keyframe from a video with javacv?

[GoogleCodeExporter]

What steps will reproduce the problem?
1.Open a video
2.Grab keyframe 
3.detect keyframe

What is the expected output? What do you see instead?
Get the keyframe in a video and save it

What version of the product are you using? On what operating system?
win7 opencv 2.4.4 javacv 0.4 eclipse 

Please provide any additional information below.
algorithm is require? 

Original issue reported on code.google.com by jizhongy...@gmail.com on 1 May 2013 at 2:11

[GoogleCodeExporter]

I suppose we would need to change the API a bit for that... Any suggestions?

Original comment by samuel.a...@gmail.com on 2 May 2013 at 2:37

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I don't know how to change the API. I want to get keyframes in the video for 
face detection.If have any class or method,please tell me.Thanks

Original comment by jizhongy...@gmail.com on 2 May 2013 at 1:47

[GoogleCodeExporter]

Let's see, it doesn't look too complicated. I made some changes in this 
revision that should do what you need:
http://code.google.com/p/javacv/source/detail?r=6547f5b5052494700cd5d19e0a8913d1
86ec735a
I haven't tested them, but it should work. Let me know how that goes! Thanks

Original comment by samuel.a...@gmail.com on 26 May 2013 at 11:28

• Changed state: Started

[GoogleCodeExporter]

year.It can be work...if can be provide a method directly,like 
'FFmpegFrameGrabber.grabKeyFrame()',The return type is 'IplImage' for playing 
keyFrame like 'canvasFrame.showImage(IplImage)',that will be easier .Of course, 
is that for me.
Thank you very much for your work!

Original comment by jizhongy...@gmail.com on 28 May 2013 at 12:51

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Yes, there is the `AVDISCARD_NONKEY` flag for that in FFmpeg.. but I don't know 
if it works, so can you try it out by setting the `AVCodecContext.skip_frame` 
field and let me know? Thanks!

Original comment by samuel.a...@gmail.com on 16 Sep 2013 at 4:06

[GoogleCodeExporter]

How ffmpeg choose key frames? I mean, what algorithm is used?

Original comment by ichar...@gmail.com on 11 Oct 2013 at 10:02

[GoogleCodeExporter]

@ichara90 They are chosen by the encoder. Here's some info about how they are 
chosen:
http://en.wikipedia.org/wiki/I-frame#Intra_coded_frames.2Fslices_.28I.E2.80.91fr
ames.2Fslices_or_Key_frames.29
But this varies greatly among codecs. Usually H.264 is best as choosing the 
most unique frames.

Original comment by samuel.a...@gmail.com on 13 Oct 2013 at 1:35

[GoogleCodeExporter]

Here's a small mod I made to ignore and skip through any non-keyframes making 
it much much faster extracting I frames:
http://pastebin.com/qZTMPzm2

Original comment by reinhard...@gmail.com on 6 Jan 2014 at 4:37

[GoogleCodeExporter]

I guess the easiest way would be to have a constructor property to make the 
whole FrameGrabber either works on all frames or I frames only.

Original comment by reinhard...@gmail.com on 6 Jan 2014 at 4:39

[GoogleCodeExporter]

`grabKeyFrame()` sounds like a good enough idea. I've added some code for that 
in the newly released JavaCV 0.7, also exposing the internal raw `AVFrame` 
objects via the `Frame.opaque` field:
http://code.google.com/p/javacv/source/detail?r=f3bb7aa577c0b427a88f927a2cea8ec5
05c31e22
Let me know if that satisfies your needs, and thanks for reporting!

Original comment by samuel.a...@gmail.com on 7 Jan 2014 at 1:19

• Changed state: Fixed

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Simple code: FFmpegFrameGrabber gr = new FFmpegFrameGrabber("video.ts"); 
gr.start(); gr.grabKeyFrame(); 
leads to multiple messages like "[h264 @ 000000001816c020] number of reference 
frames (0+5) exceeds max (4; probably corrupt input), discarding one" for 
1920*1080 H.264 file, but works good with 640*480 XVID. FrameGrabber shows both 
video ok.

Original comment by mip...@gmail.com on 9 Jan 2014 at 3:02

[GoogleCodeExporter]

Well, I guess that decoder doesn't like it when skipping packets. If you find 
the correct way to do it, please let us know, and I'll fix it, thanks!

Original comment by samuel.a...@gmail.com on 9 Jan 2014 at 2:07

[GoogleCodeExporter]

May be next info will help: when I grab this video with code
ffgr.start(); System.out.print(ffgr.getFrameRate()+" fps ");
for(i=1;i<=5;i++){ffgr.grabFrame();System.out.print(i + ";" + 
ffgr.getFrameNumber() + " | ");}
I see: "50.0 fps 1;4211359 | 2;4211361 | 3;4211363 | 4;4211365 | 5;4211367 |"
Although this video is 25 fps. And why firsе frame is 4211359 ?
With those XVID I see(using "i<=100") strange: "1;0 | 2;0 | 3;0 | 4;0 |...| 
36;0 | 37;1 | 38;1 |...| 95;22 | 96;23 | 97;23 | 98;23 | 99;24 | 100;24 |". And 
sometimes on another h264-video frames in such loop is skipped: "...26;2 | 27;2 
| 28;2 | 29;4 | 30;5 | 31;5 | 32;7 | 33;8 | 34;8 |..." 
Looks like asynchronous data update. Why grabFrame() gives frame with the same 
getFrameNumber() many times? Is this a new bug or I do something wrong? I just 
need this video [key]frame by [key]frame, no more.

Original comment by mip...@gmail.com on 9 Jan 2014 at 8:46

[GoogleCodeExporter]

getFrameNumber() computes the frame number based on the FPS and the timestamp, 
assuming a video with a non-variable fixed frame rate. If any of those 
assumptions is not true, getFrameNumber() will not return valid values. This is 
completely normal. So, please make sure that the FPS and the timestamp are 
valid, and then make sure that your video file does not have a variable frame 
rate.

If you need to know which frames are the key frames, just check the 
Frame.keyFrame flag, as explained above.

Original comment by samuel.a...@gmail.com on 11 Jan 2014 at 11:31

[GoogleCodeExporter]

Yes, now I see "frameNumber = (int)(timestamp * getFrameRate() / 1000000L);" in 
grabber code, hence it is better to avoid frameNumber value as unreliable.
Well, then if timestamp is a good value to work with, can you explain me please 
why timestamp difference between to sequential frames sometimes is <0? I 
checked 3 random movies and simple code like:
for(int i=0;;i++){ 
f=g.grabFrame();System.out.print((g.getTimestamp()-t)/1000.+", ");
t=g.getTimestamp();}
Gives: ... 40.0, 40.0, -866.489, 32.0, 32.0, 32.0, 32.0, 778.489, 40.0, 
-786.489, 32.0, 32.0... etc, while VLC-player shows all ok and inform me it is 
25 fps, so I expect always deltaT=40.0 .
The same situation with keyFrames: sometimes deltaT<0.
Are all this random HD-movies from all over the Internet really so bad 
encoded?..

As for slow checking Frame.keyFrame at every frame:
for(int i=0;;i++){ fr = gr.grabFrame(); if(!fr.keyFrame) continue; count++;}
gives completely different count=82054 than code:
for(int i=0;;i++){ fr = gr.grabKeyFrame(); count++;}
where count = 676, and it looks like true for 43min 24sec 24.98 fps movie.
While total frame counting by:
for(int i=0;;i++){ fr = gr.grabFrame(); count++;}:
gives count = 143819, which gives movie longer than 90 min at 24.98 fps.
I am confused...

Original comment by mip...@gmail.com on 13 Jan 2014 at 2:58

[GoogleCodeExporter]

I use the timestamp value returned by av_frame_get_best_effort_timestamp() so 
if there's something wrong with the value that returns, then we should let the 
FFmpeg guys know about it...

BTW, what does ffplay say aboue the timestamps?

I didn't thoroughly tested the grabKeyFrame() code. I basically use what 
Reinhard sent me above. If that doesn't work for you, please let me know how to 
fix it so that it works with your data, thank you.

Original comment by samuel.a...@gmail.com on 13 Jan 2014 at 3:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Now I am not sure about h264-timestamps at all because 
http://trac.ffmpeg.org/ticket/2910 seems to be still actual. I have 
[almost]same errors mentioned in that bug with all my h264 tested files even if 
it is re-encoded by different software like "Format factory" and Habdbrake", my 
win-build date of ffplay is Jan 9 2014, but: 
 "Cannot use next picture in error concealment
 concealing 3600 DC, 3600 AC, 3600 MV errors in P frame"
This is reproducible for any h264 file I have, and probably on any  h264 file. 
May be strange count value in my example code above ( using grabFrame() ) is 
linked to this bug.

Therefore as for grabKeyFrame(), I think now it is the only way to find 
keyframes correct for h264 and we have nothing to fix [yet?]. At least I 
compared ~100 of different movies keyframes "by my eyes" and it is the same as 
VirtualDub shows (by pressing button "show next keyframe"). Thank Reinhard for 
this working code! 
But problem with Frame.keyFrame still remains the same: code above gives 
different and very possibly incorrect counts using this frame attribute.

How to ask ffplay say about the timestamps? Sorry, I am first day ffplay user 
and could not find this in 30 minutes.

Original comment by mip...@gmail.com on 13 Jan 2014 at 2:34

[GoogleCodeExporter]

Thanks for the link!

Frame.keyFrame basically returns the content of AVFrame.key_frame, so if that's 
not correct, it's a problem with FFmpeg. Please post a bug report upstream.

As for ffplay, well in the worst case, we can check after the call to 
av_frame_get_best_effort_timestamp() in get_video_frame().

Original comment by samuel.a...@gmail.com on 14 Jan 2014 at 12:35

[GoogleCodeExporter]

Well, ffpmeg-guy answered me "the warnings were shown since forever" (see 
trac-link above). So I'd better open new bug here, because I am not too 
familiar with ffmpeg product yet to be sure what is bugs and what is just 
"features since forever".

Original comment by mip...@gmail.com on 14 Jan 2014 at 4:13

[GoogleCodeExporter]

Could also be a problem with the upstream's upstream, i.e.: x264

Original comment by samuel.a...@gmail.com on 15 Jan 2014 at 12:21