HEVC yuv420p10le HDR: out of memory

[leifhelm]

When playing back HEVC 8-bit content with mpv --hwdec=vaapi (NVD_BACKEND=direct) on an GTX 1060 I have no problems. However when playing back HDR10 content I get the following log with (NVD_LOG=1):

log

``` 1405.405771150 [21727-21738] ../src/vabackend.c:2140 __vaDriverInit_1_0 Initialising NVIDIA VA-API Driver: 10 1405.405783564 [21727-21738] ../src/vabackend.c:2149 __vaDriverInit_1_0 Now have 0 (0 max) instances 1405.405788463 [21727-21738] ../src/vabackend.c:2175 __vaDriverInit_1_0 Selecting Direct backend 1405.405869443 [21727-21738] ../src/backend-common.c: 31 isNvidiaDrmFd Invalid driver for DRM device: i915 1405.405892391 [21727-21738] ../src/direct/direct-export-buf.c: 85 direct_initExporter Found NVIDIA GPU 0 at /dev/dri/renderD129 1405.405899181 [21727-21738] ../src/direct/nv-driver.c: 246 init_nvdriver Initing nvdriver... 1405.405920483 [21727-21738] ../src/direct/nv-driver.c: 264 init_nvdriver NVIDIA kernel driver version: 550.67, major version: 550, minor version: 67 1405.405928441 [21727-21738] ../src/direct/nv-driver.c: 271 init_nvdriver Got dev info: 100 1 0 fe 1405.468799970 [21727-21738] ../src/vabackend.c:1398 nvQueryImageFormats In nvQueryImageFormats 1405.495043540 [21727-21738] ../src/vabackend.c: 637 nvCreateConfig got profile: 0 with 0 attributes 1405.495057643 [21727-21738] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 1 (8) (nil) 0 1405.495061813 [21727-21738] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 1 (8) 0x7fd9153f3980 8 1405.496599447 [21727-21738] ../src/vabackend.c:1821 nvQuerySurfaceAttributes Returning constraints: width: 48 - 4080, height: 16 - 4080 1405.496615676 [21727-21738] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 128x128, format 1 (0x7fd9153f5940) 1405.496622349 [21727-21738] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.496629051 [21727-21738] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 128x128, format 1 (0x7fd9150ec5c0) 1405.496633232 [21727-21738] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.496672170 [21727-21738] ../src/vabackend.c:2024 nvExportSurfaceHandle Exporting surface: -1 (0x7fd9150ec5c0) 1405.496677121 [21727-21738] ../src/direct/direct-export-buf.c: 188 direct_allocateBackingImage Allocating BackingImages: 0x7fd9150ef160 128x128 1405.496680984 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.496684323 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 128x128 = 16384 1405.496795581 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 128 4000 1405.496801098 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x64 (0x3) 1405.496804435 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 128x64 = 8192 1405.496940925 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 64x64 3000000004fe013 128 2000 1405.496948506 [21727-21738] ../src/direct/direct-export-buf.c: 194 direct_allocateBackingImage Importing images 1405.496953973 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 4000 1405.497089088 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 64x64 = 2000 1405.497290744 [21727-21738] ../src/vabackend.c:2037 nvExportSurfaceHandle Exporting with 128 128 0 128 3000000004fe014 0 128 3000000004fe013 1405.497648368 [21727-21738] ../src/vabackend.c: 954 nvDestroySurfaces Destroying surface -1 (0x7fd9150ec5c0) 1405.497654291 [21727-21738] ../src/direct/direct-export-buf.c: 233 destroyBackingImage Destroying BackingImage: 0x7fd9150ef160 1405.497800944 [21727-21738] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 128x128, format 100 (0x7fd9150f09a0) 1405.497807423 [21727-21738] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.497825844 [21727-21738] ../src/vabackend.c:2024 nvExportSurfaceHandle Exporting surface: -1 (0x7fd9150f09a0) 1405.497830022 [21727-21738] ../src/direct/direct-export-buf.c: 188 direct_allocateBackingImage Allocating BackingImages: 0x7fd9150f0b60 128x128 1405.497833640 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.497836956 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 256x128 = 32768 1405.497928318 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 256 8000 1405.497933779 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x64 (0x3) 1405.497937092 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 256x64 = 16384 1405.498012760 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 64x64 3000000004fe013 256 4000 1405.498017838 [21727-21738] ../src/direct/direct-export-buf.c: 194 direct_allocateBackingImage Importing images 1405.498021154 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 8000 1405.498101933 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 64x64 = 4000 1405.498203566 [21727-21738] ../src/vabackend.c:2037 nvExportSurfaceHandle Exporting with 128 128 0 256 3000000004fe014 0 256 3000000004fe013 1405.498473668 [21727-21738] ../src/vabackend.c: 954 nvDestroySurfaces Destroying surface -1 (0x7fd9150f09a0) 1405.498479347 [21727-21738] ../src/direct/direct-export-buf.c: 233 destroyBackingImage Destroying BackingImage: 0x7fd9150f0b60 1405.498591436 [21727-21738] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 128x128, format 1000 (0x7fd9150ec420) 1405.498597271 [21727-21738] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.498613658 [21727-21738] ../src/vabackend.c:2024 nvExportSurfaceHandle Exporting surface: -1 (0x7fd9150ec420) 1405.498618119 [21727-21738] ../src/direct/direct-export-buf.c: 188 direct_allocateBackingImage Allocating BackingImages: 0x7fd9150f1fc0 128x128 1405.498622385 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.498626124 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 256x128 = 32768 1405.498715910 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 256 8000 1405.498721229 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x64 (0x3) 1405.498724912 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 256x64 = 16384 1405.498789836 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 64x64 3000000004fe013 256 4000 1405.498794751 [21727-21738] ../src/direct/direct-export-buf.c: 194 direct_allocateBackingImage Importing images 1405.498798083 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 8000 1405.498876505 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 64x64 = 4000 1405.498991462 [21727-21738] ../src/vabackend.c:2037 nvExportSurfaceHandle Exporting with 128 128 0 256 3000000004fe014 0 256 3000000004fe013 1405.499264563 [21727-21738] ../src/vabackend.c: 954 nvDestroySurfaces Destroying surface -1 (0x7fd9150ec420) 1405.499269830 [21727-21738] ../src/direct/direct-export-buf.c: 233 destroyBackingImage Destroying BackingImage: 0x7fd9150f1fc0 1405.499390221 [21727-21738] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 128x128, format 4 (0x7fd9150f0860) 1405.499398289 [21727-21738] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.499421920 [21727-21738] ../src/vabackend.c:2024 nvExportSurfaceHandle Exporting surface: -1 (0x7fd9150f0860) 1405.499428152 [21727-21738] ../src/direct/direct-export-buf.c: 188 direct_allocateBackingImage Allocating BackingImages: 0x7fd9150ec4e0 128x128 1405.499433371 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.499439277 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 128x128 = 16384 1405.499534383 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 128 4000 1405.499539698 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.499542970 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 128x128 = 16384 1405.499608384 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 128 4000 1405.499613478 [21727-21738] ../src/direct/nv-driver.c: 447 alloc_image Calculated GOB size: 64x128 (0x4) 1405.499616717 [21727-21738] ../src/direct/nv-driver.c: 456 alloc_image Aligned image size: 128x128 = 16384 1405.499680770 [21727-21738] ../src/direct/nv-driver.c: 532 alloc_image created image: 128x128 3000000004fe014 128 4000 1405.499685631 [21727-21738] ../src/direct/direct-export-buf.c: 194 direct_allocateBackingImage Importing images 1405.499688951 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 4000 1405.499791059 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 4000 1405.499899301 [21727-21738] ../src/direct/direct-export-buf.c: 118 import_to_cuda importing memory size: 128x128 = 4000 1405.500022931 [21727-21738] ../src/vabackend.c:2037 nvExportSurfaceHandle Exporting with 128 128 0 128 3000000004fe014 0 128 3000000004fe014 1405.500629304 [21727-21738] ../src/vabackend.c: 954 nvDestroySurfaces Destroying surface -1 (0x7fd9150f0860) 1405.500636821 [21727-21738] ../src/direct/direct-export-buf.c: 233 destroyBackingImage Destroying BackingImage: 0x7fd9150ec4e0 1405.500871138 [21727-21738] ../src/vabackend.c: 954 nvDestroySurfaces Destroying surface -1 (0x7fd9153f5940) 1405.500881344 [21727-21738] ../src/vabackend.c: 637 nvCreateConfig got profile: -1 with 0 attributes 1405.500886160 [21727-21738] ../src/vabackend.c: 642 nvCreateConfig Profile not supported: -1 1405.533451554 [21727-21727] ../src/vabackend.c: 637 nvCreateConfig got profile: 18 with 0 attributes 1405.533465726 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) (nil) 0 1405.533471348 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) 0x1b4e000 8 1405.535408532 [21727-21727] ../src/vabackend.c:1821 nvQuerySurfaceAttributes Returning constraints: width: 144 - 8192, height: 144 - 8192 1405.535422491 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) (nil) 0 1405.535426769 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) 0x1b4e000 8 1405.537621465 [21727-21727] ../src/vabackend.c:1821 nvQuerySurfaceAttributes Returning constraints: width: 144 - 8192, height: 144 - 8192 1405.537651693 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x92bf70) 1405.537658026 [21727-21727] ../src/vabackend.c:1492 nvDeriveImage In nvDeriveImage 1405.537663707 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x92c950) 1405.537668357 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x12987e0) 1405.537672969 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x12a2d70) 1405.537677546 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x12a2f20) 1405.537682291 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x18490e0) 1405.537686629 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x18494e0) 1405.537690930 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x18498e0) 1405.537695168 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x1849ce0) 1405.537699404 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184a0e0) 1405.537703833 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184a4e0) 1405.537707979 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184a8e0) 1405.537712414 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184ace0) 1405.537716877 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184b0e0) 1405.537721214 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184b4e0) 1405.537725513 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184b8e0) 1405.537732087 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184bce0) 1405.537736595 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184c0e0) 1405.537740824 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184c4e0) 1405.537745275 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184c8e0) 1405.537749642 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184cce0) 1405.537753951 [21727-21727] ../src/vabackend.c: 920 nvCreateSurfaces2 Creating surface 3840x2160, format 100 (0x184d0e0) 1405.564273140 [21727-21727] ../src/vabackend.c: 637 nvCreateConfig got profile: 18 with 0 attributes 1405.564287533 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) (nil) 0 1405.564292066 [21727-21727] ../src/vabackend.c:1754 nvQuerySurfaceAttributes with 8 (10) 0x1928300 8 1405.565373508 [21727-21727] ../src/vabackend.c:1821 nvQuerySurfaceAttributes Returning constraints: width: 144 - 8192, height: 144 - 8192 1405.565384246 [21727-21727] ../src/vabackend.c: 984 nvCreateContext with 22 render targets, 22 surfaces, at 3840x2160 1405.820461866 [21727-21727] ../src/vabackend.c:1037 nvCreateContext CUDA ERROR 'out of memory' (2) ```

Is this expected? (mpv --hwdec=nvdec works fine for the file)

vainfo

``` Trying display: wayland Trying display: x11 libva info: VA-API version 1.20.0 libva error: vaGetDriverNames() failed with unknown libva error libva info: User environment variable requested driver 'nvidia' libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so libva info: Found init function __vaDriverInit_1_0 2056.572476965 [22587-22587] ../src/vabackend.c:2140 __vaDriverInit_1_0 Initialising NVIDIA VA-API Driver: 10 2056.572482339 [22587-22587] ../src/vabackend.c:2149 __vaDriverInit_1_0 Now have 0 (0 max) instances 2056.572486151 [22587-22587] ../src/vabackend.c:2175 __vaDriverInit_1_0 Selecting Direct backend 2056.579818703 [22587-22587] ../src/backend-common.c: 31 isNvidiaDrmFd Invalid driver for DRM device: i915 2056.579841738 [22587-22587] ../src/direct/direct-export-buf.c: 85 direct_initExporter Found NVIDIA GPU 0 at /dev/dri/renderD129 2056.579846618 [22587-22587] ../src/direct/nv-driver.c: 246 init_nvdriver Initing nvdriver... 2056.579867339 [22587-22587] ../src/direct/nv-driver.c: 264 init_nvdriver NVIDIA kernel driver version: 550.67, major version: 550, minor version: 67 2056.579872958 [22587-22587] ../src/direct/nv-driver.c: 271 init_nvdriver Got dev info: 100 1 0 fe libva info: va_openDriver() returns 0 vainfo: VA-API version: 1.20 (libva 2.21.0) vainfo: Driver version: VA-API NVDEC driver [direct backend] vainfo: Supported profile and entrypoints VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointVLD VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD VAProfileH264Main : VAEntrypointVLD VAProfileH264High : VAEntrypointVLD VAProfileH264ConstrainedBaseline: VAEntrypointVLD VAProfileHEVCMain : VAEntrypointVLD VAProfileVP9Profile0 : VAEntrypointVLD VAProfileHEVCMain10 : VAEntrypointVLD VAProfileHEVCMain12 : VAEntrypointVLD 2056.656698816 [22587-22587] ../src/vabackend.c:2050 nvTerminate Terminating 0x16da4e0 2056.656771906 [22587-22587] ../src/vabackend.c:2064 nvTerminate Now have 0 (0 max) instances ```

[thesword53]

Is your GTX 1060 a 3GB or a 6GB variant ? What version of the driver do you use ? I couldn't reproduce the issue with my GTX 1060 mobile (6GB).

[leifhelm]

My Card is a 1060 6GB (desktop). Driver version 550.67 with CUDA 12.4 according to nvidia-smi. I used these test files: https://www.demolandia.net/4k-video-test/hdr10/page-1.html?highlight=1

[thesword53]

I can reproduce the issue with my RTX 2080 SUPER and 550.67 drivers. My GTX 1060 is running with 535 drivers due to a bug with 535 drivers causing kernel crashes in laptops with integrated graphics https://forums.developer.nvidia.com/t/series-550-freezes-laptop/284772.

[elFarto]

Found the issue. It's related to use selecting Adaptive deinterlacing. This seems to consume more memory. Changing cudaVideoDeinterlaceMode_Adaptive to cudaVideoDeinterlaceMode_Weave seems to solve that issue.

However, when I do that I end up with a blue image instead of the video, so something else might be broken too.

[elFarto]

This issue should be fixed in master now.

[leifhelm]

@elFarto Thank you. Your fix works for me.