search

dubhater / vapoursynth-mvtools

Motion compensation and stuff
181 stars 27 forks source link

A way to get motion vectors out of mvtools #15

Closed haasn closed 8 years ago

haasn commented 9 years ago

I'd love to be able to use mvtools for analyzing image motion for my own postprocessing.

Would it be possible to get the results of mvtools' motion analysis out in some useful fashion? At the simplest, I'd be interested in a single “overall motion” vector, but it would also be interesting to get motion vectors for each pixel (at a smaller resolution than the video, doesn't have to be fancy).

haasn commented 9 years ago

Note: I've decided that “overall motion” is useless for my purpose, so I would be mostly interested in per-pixel motion vectors.

dubhater commented 9 years ago

The frames returned by Analyse already contain motion vectors, SAD, and who knows what else. For more information, study the source code. I think it's all in GroupOfPlanes and PlaneOfBlocks.

haasn commented 9 years ago

I'm trying to invoke this logic from within a C program; one that doesn't have anything to do with vapoursynth.

I just want some way to give it a buffer of RGB color values and get a buffer of motion vectors out.

Is this easily possible?

dubhater commented 9 years ago

You want a C API for MVTools. I wouldn't know where to begin designing one. I don't want to design one. Use VapourSynth. VapourSynth is good for you.

haasn commented 9 years ago

I've had problems with vapoursynth: It introduces unnecessary seeking latency, various hacks to be able to use it with streaming video (eg. in a video player) rather than a static file, and the performance of vapoursynth-mvtools is also unusably slow because FlowFPS runs on the CPU.

I want to get the motion vectors out of it and do my own interpolation on the GPU so it can be done in realtime instead of offline.

Also, vapoursynth integration is a gigantic mess that's flaky even in mpv's vf_vapoursynth, in particular due to its asynchronous nature which is not very ideal for realtime, immediate playback.

Even if I were to use vapoursynth somehow, I can't just set the motion vectors clip as the output, can I? I have to get this information out to the C program somehow so I can upload it to the GPU and do my own computations.

dubhater commented 9 years ago

You can set the motion vectors clip as the output. It won't work in vspipe because vspipe doesn't like clips with unknown dimensions, but your C program doesn't need to care about that. The data is in the frames returned by mv.Analyse. You just call getReadPtr() and you have it. (In the future, I'd like it to return the input frames unchanged, with the motion vector data attached as a frame property, but that won't change much for you.)

dubhater commented 8 years ago

Can this be closed yet? :p

haasn commented 8 years ago

Well, if the status quo on a C API is “won't ever implement”, then feel free to close.

dubhater commented 8 years ago

Weeeell, technically, it now has a C API internally. But I'd rather not expose it because that brings additional work. :)

haasn commented 8 years ago

Well, that's fully up to you. I don't have a very strong interest in this feature. I might play around with a C API if you end up exposing one, but I don't care enough to work my way through the vapoursynth abstraction just for the sake of an experiment.

I'd be more likely to write my own motion estimation algorithm on the GPU :p