Remove FFMPEG dependency and use GStreamer instead

nvpro-samples / vk_video_samples

Vulkan video samples

Apache License 2.0

239 stars 40 forks source link

Remove FFMPEG dependency and use GStreamer instead #22

Open dabrain34 opened 1 year ago

dabrain34 commented 1 year ago

I'm planning to add GStreamer capabilities to replace the FFMpeg parser in https://github.com/nvpro-samples/vk_video_samples/blob/de54a1df626828c47e265c65d6676a3f083809a1/vk_video_decoder/demos/vk-video-dec/AppDecVulkanFrame/VulkanVideoProcessor.cpp#L49

Do you prefer an option and keep FFMpeg or just like drop FFMpeg for real ?

dabrain34 commented 1 year ago

See https://github.com/nvpro-samples/vk_video_samples/pull/21

zlatinski commented 1 year ago

Do you prefer an option and keep FFMpeg or just like drop FFMpeg for real ?

I would still like to keep the FFMPEG as an option, please, since we still use it as video demuxer with the NV parser.

dabrain34 commented 1 year ago

no problem at all to keep the 2. Its just it would be easier to have one library for all.

BattleAxeVR commented 1 year ago

I was looking into GStreamer and it seems like a good alternative to FFMPEG, so I would like to see this PR merged to the main branch at some point too. Anyway the main branch doesn't even work at all right now

dabrain34 commented 1 year ago

We could also use https://github.com/Igalia/ESExtractor

BattleAxeVR commented 1 year ago

I would kill to find a video decoder that can also output the motion vectors. That's super useful for using video streams in videogames, if you're using things like DLSS. FFMPEG CPU decoder is supposed to have such a feature but that was ages ago I don't think was ever merged.

zlatinski commented 3 months ago

Hi @BattleAxeVR, Can you please try https://developer.nvidia.com/optical-flow-sdk? We are still considering a Vulkan video extension that will output the motion vectors, but I'm not sure if this extension will be done this year.

BattleAxeVR commented 3 months ago

Thanks for the tip, I noticed that project a while back and how it would be suitable, however I've already implemented a working pipeline for vulkan video with explicit MVs stored in a separate video file.

I mostly needed the MVs as inputs to DLSS, which I've discovered can be exploited to produce some rather neat special FX like mirages or optical lensing / hot air shimmering the scene behind it: