search

aizvorski / h264bitstream

A complete set of functions to read and write H.264 video bitstreams, in particular to examine or modify headers.
GNU Lesser General Public License v2.1
732 stars 238 forks source link

Extract motion vectors #38

Open adrianmay opened 4 years ago

adrianmay commented 4 years ago

Hi there,

I'm looking for a very fast way to extract motion vectors from an h264 stream inside some container. I just need the x,y coordinates of the block and it's dx,dy movement. I'd like to cut out any CPU time not contributing to the motion vectors. The data is coming from ffmpeg (encoding as h264 in hardware) so I can set the container format to anything.

Is this repo what I'm after?

Please excuse my ignorance of h264 but I can't answer this question by looking at the code, nor would I know how to do it if you just said "yes", so some guidance would be very much appreciated.

I know this isn't really an "issue" but github seem to have disabled all other forms of messaging.

TIA, Adrian.

aizvorski commented 4 years ago

Hi Adrian - Thanks for asking. The MVs are contained in the slice data - decoding is not currently supported, as this library mostly works with header data. There is some work-in-progress code, please see #8. If you want to dig into that, that would be great, but it is going to be quite a lot of work - mainly because this code doesn't have working CABAC (yet).

There are some possibly better starting points:

Note that in addition to the MVs, you will at a minimum need the macroblock types, partitioning modes and likely the reference frames as well. It is simpler than fully decoding the video (no DCT, no motion compensation) but not hugely simpler.

I'm interested in this as well, please let me know how you solve this.

lbdroid commented 4 years ago

I believe that there would be a huge demand for this feature if it were to be implemented. In particular, I'd like to be able to apply this to motion detection in the field of physical security. Motion detection in open source software currently requires full decompression and frame-by-frame analysis, and the CPU cost is near prohibitive.

If motion vectors could be extracted from the compressed domain, it would allow high resolution and high frame rate motion detection even on very low end hardware.

aizvorski commented 4 years ago

@lbdroid Indeed, that would be useful. I can offer to do this feature as a bounty, as it involves a significant effort - please contact me at aizvorski at gmail if you're interested.