search

LukasBommes / mv-extractor

Extract frames and motion vectors from H.264 and MPEG-4 encoded video.
MIT License
279 stars 56 forks source link

Missing blocks in motion-compensated frame #21

Open zohrehazizi opened 2 years ago

zohrehazizi commented 2 years ago

Hi, Thank you for developing this great tool for motion extraction.

I used the tool to extract the motion vectors in the example video. Let's say we have the reference frame, the current frame, and the motion vectors corresponding to current frame.

I used the motion vectors to apply motion compensation on the reference frame in order to reconstruct the current frame. After motion compensation, there are some empty macro blocks left in the frame. They are the macro blocks which no motion vector was pointed to in the current frame. I suppose that for these macro blocks, no similar macro block in the reference frame was found.

Now I have two questions. Is there anywhere in the code that I can set the threshold for finding the similar Macroblock. I kind of hope to have less empty Macroblocks in the motion compensated frames. My second question is that is it possible that we have bi-directional frames using this tool? I think if we have bi-directional frames, there would be less empty macrobloacks in the motion-compensated frame.

I appreciate your response. Thank you!

Reference frame: frame-0

Current frame: frame-1

motion-compensated reference frame: frame-1-mvComp

LukasBommes commented 2 years ago

Hey zohrehazizi, glad to hear that you find mv-extractor useful. To answer your questions:

1) Is there anywhere in the code that I can set the threshold for finding the similar Macroblock. I kind of hope to have less empty Macroblocks in the motion compensated frames.

-> This has to be done during encoding of the video using e.g. FFMPEG. With FFMPEG you can either transcode the video or split it up into individual frames and encode them as a video again. I am not sure if the codec directly supports a threshold as you need it. But I assume the compression level and profile (baseline, main, high) has an impact on the number of fully coded (i.e. not motion-compensated) macroblocks.

2) My second question is that is it possible that we have bi-directional frames using this tool? I think if we have bi-directional frames, there would be less empty macrobloacks in the motion-compensated frame.

Yes, B-frames are supported. But again, creating B-frames is part of the encoding procedure. The H.264 profile determines whether b-frames are available.