search

LukasBommes / mv-extractor

Extract frames and motion vectors from H.264 and MPEG-4 encoded video.
MIT License
305 stars 60 forks source link

MV extraction for blocks using intra frame reference #13

Closed Schwartz-Zha closed 2 years ago

Schwartz-Zha commented 2 years ago

Great work first, much thanks to the author, saving a lot of time coding in c.

But I am not really sure about the mechanism here. And I do not know if the function of this project is working as expected. For example, my purpose is to use this motion vector to partially replace the optical flow process in video processing. And I generate my video using ffmpeg with exactly the following command:

ffmpeg -r 60 -f image2 -s 1280x720 -i %08d.png -bf 0 -refs 1 -vcodec libx264 -crf 25 -pix_fmt yuv420p test.mp4ffmpeg -r 60 -f image2 -s 1280x720 -i %08d.png -bf 0 -refs 1 -vcodec libx264 -crf 25 -pix_fmt yuv420p test.mp4

Two settings are worth noting here, I have removed the B frames and set the reference frame number of P frame to only 1, so basically this setting is very close to optical computation (each frame is only referenced from the previous frame).

However, I found that the generated video uses both inter and intra macroblocks for compression. Say, in a frame, two macroblocks are numerically identical. It is reasonable to use one to reference another, leaving only a motion vector to save. But this motion vector does break the common understanding of "motion" as this process has nothing to do with temporal movement. It is simply a strategy to save some memory within one frame.

I want to confirm with the author if this situation is considered when extracting motion vectors. Any suggestion, please shoot me.

LukasBommes commented 2 years ago

Dear Schwartz-Zha,

thanks for the great question. I honestly do not recall the details on how intra macroblocks are handled. It would be great, if you could just try out how mv-extractor deals with them and share here what you found.

Anyway, the motion vectors used for compression should correlate reasonably well with the scene motion (at least they did for the footage I tested on). I used the motion vectors for multi-object tracking and achieved good tracking accuracy. Most likely you would get more accurate motion estimates by computing optical flow. But it is also slower than using the motion vectors provided by the codec.

Regards, Lukas

LukasBommes commented 2 years ago

Closing due to inactivity. Feel free to reopen for further discussion.