search

RomanArzumanyan / VALI

Video processing in Python
Apache License 2.0
21 stars 1 forks source link

Does VALI support GetMotionVectors method? #13

Closed cheptil closed 5 months ago

cheptil commented 5 months ago

Hello

In VPF there is GetMotionVectors method, does VALI support this method?

RomanArzumanyan commented 5 months ago

@cheptil

Please check out issue-13_get_mv branch, it has the fix for your issue. Here's the API usage example: https://github.com/RomanArzumanyan/VALI/blob/9eb701aba840e979477e6dc59991f22923deca4f/tests/test_PyFFmpegDecoder.py#L171-L192

cheptil commented 5 months ago

@RomanArzumanyan

Thank you very much for the update!

I have tested it with test_res_change.h264 example from tests and see that num_elems for second frame was not zero, but the mv array is empty

Is it some bug or I need get access to mv info in another way?

RomanArzumanyan commented 5 months ago

Hm, that’s interesting.

I don’t observe such behaviour on basic test video, let me check the resolution change.

RomanArzumanyan commented 5 months ago

@cheptil

After bashing against custom numpy array types for a while I've switched PyFfmpegDecoder.GetMotionVectors method return data type to List[nvc.MotionVector].

Looks like it fixes the problem, I've tested both basic and resolution change videos.

cheptil commented 5 months ago

@RomanArzumanyan

Great! Thank you a lot, I will test it tomorrow and write here the result

Also I have a question, does this implementation support receiving motion vectors for hevc?

RomanArzumanyan commented 5 months ago

@cheptil

This method uses FFMpeg API for side data retrieval. So it supports everything FFMpeg does.

cheptil commented 5 months ago

@RomanArzumanyan

I have tested new change and confirm that now GetMotionVectors method works!

Also I noticed that for hevc video this method return empty mv array

As far as I understand, VPF/VALI do not use Video Codec SDK, so there is no AppEncME mode, only FFMpeg API — am I right?

RomanArzumanyan commented 5 months ago

Hi @cheptil

I have tested new change and confirm that now GetMotionVectors method works!

Good news, thanks.

Also I noticed that for hevc video this method return empty mv array

If VALI returns empty MV array for HEVC P or B frame please provide me with repro video, I'll check on my side. Motion Vectors API retrieval is codec-agnostic so it shall work for H.264 / HEVC etc.

As far as I understand, VPF/VALI do not use Video Codec SDK, so there is no AppEncME mode, only FFMpeg API — am I right?

VALI does rely on Video Codec SDK for HW-accelerated video enc / dec. It also uses FFMpeg for demuxing video containers and providing CPU-accelerated decoder fallback. It may be useful when input isn't supported by GPU decoder or simply if you want to utilize all the HW you have.

cheptil commented 5 months ago

@RomanArzumanyan

There is an example of hevc video (please unzip it)

sample_960x540.zip

RomanArzumanyan commented 5 months ago

Hi @cheptil

After inspecting the code in debugger I found that FFMpeg doesn't extract motion vectors for your H.265 video, hence VALI returns you an empty array.

Most probably I'm mistaken in my assumption that FFMpeg allows to extract MV for variety of codecs. Looks like it only supports H.264. I've found similar questions on SO and ffmpeg mailing list:

Also there's a MV extraction patch for HEVC which to my understanding wasn't merged to FFMpeg codebase:

Long story short: I'm afraid PyFFmpegDecoder only allows you to get MV for H.264 videos as that's what FFMpeg supports under the hood.

P. S. Please notice that ME only Nvenc mode doesn't extract MV from compressed video file. It runs it's own ME algo under the hood and return you it's estimations.

RomanArzumanyan commented 5 months ago

@cheptil

I need to merge linked PR, it will close this issue. Please feel free to re-open.