Add support for NVIDIA Optical Flow Accelerator (NVOFA) FRUC

[OpenSourceAnarchist]

Expected behavior of the wanted feature

MPV already ships with CUDA support I believe. All this would require, then, is adding in libraries for NVIDIA Optical Flow SDK 4.0, which includes the NVOFA assisted Frame Rate Up Conversion (FRUC) library. The library exposes FRUC APIs that can be used for frame rate up conversion of video files. See https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvfruc-programming-guide/index.html for the API documentation and sample code.

This is important because NVIDIA GPUs since the Turing generation have hardware support for optical flow-based frame interpolation. MPV already includes --interpolation, but if the GPU supports optical flow-based frame interpolation, this should obviously be preferred for maximum smoothness and quality. I believe NVOFA FRUC only returns interpolated frames, so exactly which frames are interpolated would need to be decided (to match the user's monitor's frame rate, to match to an arbitrary fps, to just interpolate between all frames --> 2x fps, etc.).

Alternative behavior of the wanted feature

Even a "dumb" implementation that simply interpolates all frames (2x total frames) would be highly appreciated. Effectively, this would eliminate the need to use SVP since MPV itself would be capable of hardware-based frame interpolation for NVIDIA GPU owners!

[philipl]

This is something I've been thinking about doing, but two things:

• It would be an FFmpeg filter, and not actually any code in mpv
• It would almost certainly count as nonfree in FFmpeg and so be unredistributable in binary form if it was even considered acceptable to merge.

The api itself says it can only accept nv12 and rgb, which would make proper handling of 10/12 bit content impossible, as you really don't want to do interpolation on the final scaled rgb and neither do you want to do rgb conversion in a filter as opposed to in the vo.

[OpenSourceAnarchist]

Thanks for clarifying those points. Should I make a bug report on FFmpeg's tracker and link it here since this issue can't be fulfilled until the filter gets upstreamed? I can mention the caveat with 10/12 bit video files.

I guess the only code in mpv needed would then be a build flag to include support for NVOFA FRUC and a runtime flag to enable/disable it (and perhaps what GPU device you want it to run on). Or since it's just a FFmpeg filter, it could be left to the user to manually specify the filter and any parameters which would be passed along to FFmpeg.

[philipl]

There's no point creating an ffmpeg bug. I know that I'm trying this out and no one else will pay any attention to it.

Anyway, the whole thing is horribly written with poor documentation. It currently segfaults in an undebuggable way and I'm seeing if I can get any help from nvidia to understand why it's not working.

[philipl]

Ok. I've got a working implementation based on vf_framerate (which is a simpler blending filter). As noted, the licensing here is dubious - I do not believe any resulting binaries are distributable, and you need to provide libNvOFFRUC.so and libcuda-rt in your library path. But feel free to try it out.

mpv --vf=nvoffruc:fps=120 somefile.mkv

[philipl]

I've cleaned up the change on my branch. Feel free to try it out. It's never going to be mergeable given the licensing, but you can build it and use it locally if you want. I'm going to close this issue as that's the closest you're going to get under the current circumstances.

[Usulyre]

I've cleaned up the change on my branch. Feel free to try it out. It's never going to be mergeable given the licensing, but you can build it and use it locally if you want. I'm going to close this issue as that's the closest you're going to get under the current circumstances.

Hi a bit off topic here, but do you have a ffmpeg binary to try out the filter with? Also, how does one specify the parameters in ffmpeg for the filter?

[Jules-A]

AMD released the FSR3 source code, I wonder if it would be more practical to support the interpolation from that since it wouldn't have license issues and would support all vendors. EDIT: It looks like it only supports DX12 currently so it would need to be ported :/

[philipl]

Hi a bit off topic here, but do you have a ffmpeg binary to try out the filter with? Also, how does one specify the parameters in ffmpeg for the filter?

No, you'll need to build your own binary. As I said up top, the binaries cannot be distributed due to licence incompatibility - you must build your own binaries.

Parameter passing follows the normal documented ffmpeg filter syntax.

[philipl]

AMD released the FSR3 source code, I wonder if it would be more practical to support the interpolation from that since it wouldn't have license issues and would support all vendors. EDIT: It looks like it only supports DX12 currently so it would need to be ported :/

Someone will probably poke around with it eventually but it's not clear to me if it can work without the detailed motion vectors that a game can produce (because it knows exactly where each object in the scene is going). For decoded video, we don't necessarily have any motion vectors, and even for codecs with such vectors, they are tracking where macroblocks are going and not scene elements.

[Jules-A]

Someone will probably poke around with it eventually but it's not clear to me if it can work without the detailed motion vectors that a game can produce (because it knows exactly where each object in the scene is going). For decoded video, we don't necessarily have any motion vectors, and even for codecs with such vectors, they are tracking where macroblocks are going and not scene elements.

Yeah, I don't know about that either but I did notice this recently added to AMF: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/c48e50ad6c8723c006b2c145d8fa49ecc0651022/amf/doc/AMF_FRC_API.pdf

Not sure how similar it would be to FSR3 but it looks interesting.

[fideliochan]

@philipl How can I make it do container_fps*2 Also do I have to enable display-resample, tscale and interpolation. I asked it since we pass it as vf

[Jules-A]

@philipl How can I make it do container_fps*2 Also do I have to enable display-resample, tscale and interpolation. I asked it since we pass it as vf

Pretty sure MPV will just see the video with twice the framerate of the original and none of those options are required. Although I am curious to how interpolation works when combined.

[philipl]

Right. This filter can do any arbitrary target frame rate, so you shouldn't need to use any other mechanisms. Eg: you can do 24->60 with it and, in theory, you will get proper 60fps frame pacing. But if you do decide to 24->48 plus interpolation, it would work fine too.

As for container fps substitution, I think that requires additional logic. I can look at it, but no promises.

[philipl]

I have added support for source_fps expressions.

[philipl]

Actually, I'm reopening this just to improve visibility for anyone who wants to use this.

[fideliochan]

@philipl Thank you for creating this fork. I have no experience with compiling, but I followed the steps of ./configure, make, and sudo make install. However, I noticed that it did not contain vf_nvoffruc. I tried again with ./configure --enable-nonfree, but now I am getting an error with nvoffruc.h. I downloaded Optical Flow_SDK5.0.7.zip and found the NvOFFRUC.h file in Optical_Flow_SDK_5.0.7/NvOFFRUC/Interface. Furthermore, I copied and pasted it into FFmpeg's folder and tried again, but I am still encountering the same issue. Could you please advise me on what I might be doing wrong?

[philipl]

I cannot provide a detailed guide on how to ensure you have all the required dependencies, but for this specific case, you must place NvOFFRUC.h in the libavfilter/ directory next to the filter code.

You must have https://github.com/FFmpeg/nv-codec-headers installed as well, if you haven't already (and maybe your distro has a package for it)

[Usulyre]

I cannot provide a detailed guide on how to ensure you have all the required dependencies, but for this specific case, you must place NvOFFRUC.h in the libavfilter/ directory next to the filter code.

You must have https://github.com/FFmpeg/nv-codec-headers installed as well, if you haven't already (and maybe your distro has a package for it)

I am trying to cross compile this using this script here:

https://github.com/rdp/ffmpeg-windows-build-helpers

Does NvOFFRUC.h have to be modified in some way? Or do I just place it in the libavfilter folder?

Do I need any other files to placed in any certain folders?

I just can't find any compiling instructions here for this.

I have installed the nv-codec-headers as well.

Anything else to add to make this compile correctly?

Thanks in advance.

[philipl]

Yes, you do need to modify it, as documented in the code: https://github.com/philipl/FFmpeg/blob/nvoffruc/libavfilter/vf_nvoffruc.c#L42

[Usulyre]

Yes, you do need to modify it, as documented in the code: https://github.com/philipl/FFmpeg/blob/nvoffruc/libavfilter/vf_nvoffruc.c#L42

Ok I did follow that.

Two steps?

Remove using namespace std; Replace bool * with `void *'

but still getting errors like below.

`In file included from libavfilter/vf_nvoffruc.c:48: libavfilter/NvOFFRUC.h:215:5: error: expected '=', ',', ';', 'asm' or 'attribute' before '{' token 215 | { | ^

In file included from libavfilter/vf_nvoffruc.c:48: libavfilter/NvOFFRUC.h:244:9: error: unknown type name 'NvOFFRUCResourceType' 244	NvOFFRUCResourceType eResourceType; /*< [in]: Specifies whether resource created by client is DX or CUDA. /	^~~~~~~~ libavfilter/NvOFFRUC.h:245:9: error: unknown type name 'NvOFFRUCSurfaceFormat' 245	NvOFFRUCSurfaceFormat eSurfaceFormat; /*< [in]: Specifies surface format i.e. NV12 or ARGB. /	^~~~~ libavfilter/NvOFFRUC.h:246:9: error: unknown type name 'NvOFFRUCCUDAResourceType' 246	NvOFFRUCCUDAResourceType eCUDAResourceType; /*< [in]: Specifies whether CUDA resource is cuDevicePtr or cuArray. /	^~~~~~~~ libavfilter/vf_nvoffruc.c: In function 'process_work_frame': libavfilter/vf_nvoffruc.c:291:58: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'int64_t' {aka 'long long int'} [-Wformat=] 291	av_log(ctx, AV_LOG_DEBUG, "Matched f0: pts %lu\n", work_pts);	^ ~~
	int64_t {aka long long int}
long unsigned int
%llu

libavfilter/vf_nvoffruc.c:294:58: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'int64_t' {aka 'long long int'} [-Wformat=] 294	av_log(ctx, AV_LOG_DEBUG, "Matched f1: pts %lu\n", work_pts);	^ ~~
	int64_t {aka long long int}
long unsigned int
%llu

libavfilter/vf_nvoffruc.c:297:57: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'int64_t' {aka 'long long int'} [-Wformat=] 297	av_log(ctx, AV_LOG_DEBUG, "Unmatched pts: %lu\n", work_pts);	^ ~~
	int64_t {aka long long int}
long unsigned int
%llu

make: *** [ffbuild/common.mak:81: libavfilter/vf_nvoffruc.o] Error 1`

[philipl]

That suggests you might have mangled something. Line 215 in the original flle in innocuous. make sure you didn't actually delete something else.

[Usulyre]

That suggests you might have mangled something. Line 215 in the original flle in innocuous. make sure you didn't actually delete something else.

OK.

Well i got this error originally:

In file included from libavfilter/vf_nvoffruc.c:48: libavfilter/NvOFFRUC.h:25:14: fatal error: Windows.h: No such file or directory 25 | #include

However after a bit of research I tried to rename a file "windows.h" to uppercase which leads to those errors i guess? It's in the cross compilers that the script I have uses.

Or is it something else?

[philipl]

I'm afraid I don't know enough about the windows build to help with that. Certainly you should modify NvOFFRUC.h rather than rename the file - which might cause problems elsewhere. But windows is supposed to be case insensitive, so it shouldn't make a difference.

[Usulyre]

Ok. I've got a working implementation based on vf_framerate (which is a simpler blending filter). As noted, the licensing here is dubious - I do not believe any resulting binaries are distributable, and you need to provide libNvOFFRUC.so and libcuda-rt in your library path. But feel free to try it out.

mpv --vf=nvoffruc:fps=120 somefile.mkv

Do I need "libNvOFFRUC.so and libcuda-rt in my library path" for compilation?

[philipl]

No, but unfortunately, I am reminded of https://github.com/philipl/FFmpeg/blob/nvoffruc/libavfilter/vf_nvoffruc.c#L318

There is no logic for doing dynamic symbol loading on Windows.It's not particularly difficult as you can use compat/w32dlfcn.h to probably make the existing dlopen and dlsym work as-is. But you'll need to do it. Sorry.

[Andarwinux]

Just use lld and delayload.

[Usulyre]

No, but unfortunately, I am reminded of https://github.com/philipl/FFmpeg/blob/nvoffruc/libavfilter/vf_nvoffruc.c#L318

There is no logic for doing dynamic symbol loading on Windows.It's not particularly difficult as you can use compat/w32dlfcn.h to probably make the existing dlopen and dlsym work as-is. But you'll need to do it. Sorry.

Not sure once I get around those strange errors, that I understand what to do for dynamic symbol loading.

Any good help would be appreciated.

[SephirothFFKH]

I think another approach was the vapoursynth route which reminds me of an old project with the legacy NV Optical Flow vapoursynth-nvof but the maintainer @mystery-keeper never upgraded that project to NV OFFRUC unfortunately.

[philipl]

I think another approach was the vapoursynth route which reminds me of an old project with the legacy NV Optical Flow vapoursynth-nvof but the maintainer @mystery-keeper never upgraded that project to NV OFFRUC unfortunately.

The vapoursynth integration doesn't support passing hardware frames so the overhead of copying back and forth between system and GPU memory multiple times rather undermines the whole approach.

[Usulyre]

No, but unfortunately, I am reminded of https://github.com/philipl/FFmpeg/blob/nvoffruc/libavfilter/vf_nvoffruc.c#L318

There is no logic for doing dynamic symbol loading on Windows.It's not particularly difficult as you can use compat/w32dlfcn.h to probably make the existing dlopen and dlsym work as-is. But you'll need to do it. Sorry.

I'm still having issue with the NvOFFRUC.h file can you help out with that?