search

LukasBommes / mv-extractor

Extract frames and motion vectors from H.264 and MPEG-4 encoded video.
MIT License
304 stars 60 forks source link

About the motion vector #16

Closed JXH-SHU closed 2 years ago

JXH-SHU commented 2 years ago

Thank you so much for open-sourcing your code! But I used the provided environment to print the motion vector directly and found that it is not complete, (There is no corresponding motion vector for some areas). Looking forward to your reply!

LukasBommes commented 2 years ago

Hey JXH-SHU, could you provide a screenshot of the motion vectors output by mv-extractor? Depending on the visual content it is normal that some motion vectors are close to zero and consequently not visible. Take a look at the title image in the readme. Many vectors are actually zero or close to zero. It just means that the macro block associated to the motion vector hasn't moved since the last frame.

JXH-SHU commented 2 years ago

Thank you for your reply. I printed the motion vector and block division maps. I found that there is no motion vector where the motion is severe in the two images mv_arrow mv_block .

LukasBommes commented 2 years ago

Without seeing the entire sequence, I would say the motion vector field looks quite reasonable. The camera seems to tilt horizontally, which is why the entire background moves horizontally. The players probably move in such a way that they stay in the same location between the frames. Hence, the motion vectors are of small magnitude.

In any case, mv-extractor just extracts what is provided in the video. You can also try to transcode into MPEG-4. If you first split your videos into individual frames, you can encode them into an MPEG-4 video with the following command:

ffmpeg -y -r <f> -i %06d.jpg -c:v mpeg4 -vf 'pad=ceil(iw/2)*2:ceil(ih/2)*2' -f rawvideo out.mp4

where is the frame rate. (You can also transcode directly, but you'll have to figure out he exact ffmpeg command)

JXH-SHU commented 2 years ago

Thanks again for your reply. In fact, I also extract data from a mp4 video. Here, I added three consecutive frames. What confuses me is that there is no corresponding optical flow in places with intense motion. img000084 img000085 img000086

LukasBommes commented 2 years ago

To me, the motion vectors still look reasonable. At least in the three frames you posted above, there is not much motion around the players, whereas the entire background moves to the left.

However, I am not entirely sure, why the macroblock segmentation is not present in the area of the players. Maybe, the respective area is not coded via motion prediction but instead fully coded. I can't really remember if that is something that is done in H.264.

JXH-SHU commented 2 years ago

Thank you very much for your reply.