I couldn’t use AVPro to play video with NVDEC

[keitasumiya]

Describe the bug I couldn’t use AVPro to play video with NVDEC.

This issue is relevant to the following unsolved issue: https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/86

My final goal is the same as the above issue and it is playing 8K 4:4:4 H.265 60fps video with AVPro. But I would like to solve a more fundamental problem that is using AVPro with NVDEC.

I checked some combinations about resolution, fps, codec, chroma subsampling and player(AVPro, FLVPlayer and MPC-BE).

• The result spreadsheet is here: https://docs.google.com/spreadsheets/d/1-tIofJkbMChZdvT45-2O5k5_tWz9SRJkRUKd49Ml6NA/edit?usp=sharing or attached pngs (avpro.png, flv.png, mpcbe.png) in the below Screeanshots.

• Each screenshots are here: https://www.dropbox.com/sh/ejml7m4ddywc0vk/AAAnrkWlbG2Ir0qezPjoXutra?dl=0

• 32 Videos I used are here: https://www.dropbox.com/sh/598ni0i4wf92w9b/AACJCjhUXLwENTWhEEUlI8lGa?dl=0

As a result, I understood it seems that AVPro doesn’t use hardware decoder for Quadro RTX4000 at least. As written in NVIDIA’s page: https://developer.nvidia.com/video-encode-decode-gpu-support-matrix Quadro RTX4000 with NVDEC is available until 8K 4:4:4 H.265 video. Moreover, AVPro sometimes recommends using LAV filter in an official document: http://downloads.renderheads.com/docs/UnityAVProVideo.pdf

Therefore, I guessed that AVPro doesn’t use NVDEC by default. Then I searched for a way to use AVPro with NVDEC, but I couldn’t find it.

This is the reason why I ask this issue here.

Your Setup (please complete the following information):

• Unity version: 2018.4.1f1
• AVPro Video version: 1.9.17
• Operating system version: Windows 10 pro
• Device model:
  • desktop pc
  • Intel xeon E5-2680 v4
  • memory 128GB
  • GPU: NVIDIA Quadro RTX4000
  • GPU driver: 431.70
• Video specs (resolution, frame-rate, codec, file size):
  • filename extension: mp4
  • resolution: 1K, 2K, 4K, 8K
  • fps: 30, 60
  • codec: H.264, H.265
  • chroma subsampling: 4:2:0, 4:4:4
  • file size: 168MB/60sec for 8K-60fps-60sec-h265-444.mp4

To Reproduce

1. Drag and drop "Assets/AVProVideo/FullscreenVideo" into hierarchy window
2. select "FullscreenVideo" in hierarchy window
3. select "Platform Specific"
4. select "Windows"
5. select "Direct Show" at Preferred Video API
6. select "BROWSE" to change a video
7. select my video (8K-60fps-60sec-h265-444.mp4 etc.)
8. Unity Play

Logs

Screenshots Checked tables are the followings:

Each screenshots are here: https://www.dropbox.com/sh/ejml7m4ddywc0vk/AAAnrkWlbG2Ir0qezPjoXutra?dl=0

Videos

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

[AndrewRH]

Hi,

Thanks for so many details! Before I dive in, can I ask whether you tried the Media Foundation video API option instead of changing it to DirectShow? On DirectShow path we don't have any GPU decoding support, but with Media Foundation it does.

Thanks,

[AndrewRH]

Does that help?

[keitasumiya]

I am sorry for my late reply!! I tried both Media Foundation and DirectShow. But, when I had tried with Media foundation, I couldn't play video.

[AndrewRH]

Couldn't even play H.264?

For H.265 you may have to install the Codec from Microsoft here: https://www.microsoft.com/en-us/p/hevc-video-extensions-from-device-manufacturer/9n4wgh0z6vhq

[keitasumiya]

Thank you for your advice!!

Yes, I couldn't play the video (H.264 and color 4:4:4) with AVPro via Media Foundation.

And, for H.265, I had already installed the codec from Microsoft but it didn't work.

[AndrewRH]

That's strange...do you have a K or N edition of Windows?

These editions don't include any codecs and require installing the Media Pack from Microsoft. https://www.microsoft.com/en-us/download/details.aspx?id=48231&WT.mc_id=rss_alldownloads_all

Also make sure you have latest GPU drivers - and don't install Geforce Experience.

Does any of that help?

[keitasumiya]

Thank you for your reply. I'll check it and reply again, so please wait for it!!

[keitasumiya]

I am so sorry for my late reply. My windows is

• edition : Windows 10 Pro
• version : 1903 .

So I think my windows is not K or N edition.

How do you think it?

[AndrewRH]

Hmm..that's strange...could you try on another PC?

Our demo scenes use H.264 videos and they should all work....

Haven't heard of an issue like this before, but it would be nice to work out why it's not working in your case.

[keitasumiya]

Thank you for your reply!! OK, I’ll try this problem in another pc, so is it ok if you wait it?

[AndrewRH]

Did you have any luck? I think we'll have to close this ticket/issue soon as we're not sure what's wrong and nobody else has reported this issue....

[keitasumiya]

Hello again!! This problem is not solved yet... I tried another devices but it doesn't work.

Recently, I try to play some videos like:

• 8K(7680x4320)
• 60fps
• hevc
• 4:2:0

but it also doesn't work...

One of my devices I tried is the followings: pc:

• OS: windows 10 pro for workstations
• CPU: intel xeon(R) E-2176M CPU@2.70GHz
• memory(RAM) : 64GB
• system: 64bit
• GPU: NVIDIA Quadro P2000 with Max-Q Design
• Lenovo, Thinkpad P1

If you are OK, please give me some advices to solve it.

[keitasumiya]

@AndrewRH I am so sorry for my trouble. If you have a time, I would like your team's advices. I am looking forward to getting your reply!! :)

[AndrewRH]

Hi,

I created some 8K60 HEVC videos but they all worked fine in my setup. One thing I noticed that's weird in your mediainfo is that your video is set to profile+level Main6.2. This level should be for videos beyond 8K60, like 8K120 etc....So I think perhaps your encoder has a bug. I'm using ffmpeg to encode my videos and it's selecting the Main6.1 profile which is correct. You can read about these profiles/levels here: https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding_tiers_and_levels

Here is the FFMPEG command I'm using as a test to create my test videos:

ffmpeg -hide_banner -y -i %1 -pix_fmt yuv420p -r 60 -vf scale=7680x4320,setsar=1:1 -c:v libx265 -vtag hvc1 -tune fastdecode %1-hevc-8k60.mp4

So perhaps try using FFMPEG, or try changing your version of FFMPEG if you're already using it?

I'm testing on a laptop too - With 2 GPU's Intel and NVidia Geforce 1050. Your machine also seems to have 2 GPU's. I would recommend going into the NVidia Control Panel and making sure the NVidia is set as the default GPU instead of Auto Detection.

If you're using HEVC then you may also need to install the Microsoft HEVC codec: https://www.microsoft.com/en-us/p/hevc-video-extensions-from-device-manufacturer/9n4wgh0z6vhq?activetab=pivot:overviewtab

If it still doesn't work - try encoding without any audio, perhaps it's causing some problems.

Here you can see it running on my laptop:

Note that you can see from the Task Manager that my Geforce 1060 video decoder is running at 100% so with the extra overhead of Unity and rendering etc you're really running dangerously close to the system limits and may drop frames.

I hope this helps?

[keitasumiya]

Thank you so much for your kind reply!!

I perfectly agree your advice. Maybe my encoder is one of big reasons about this problem. Until now, I usually used ffmpeg, premiere pro, davinci resolve etc. However, AVPro doesn't use the "hardware Video Decoding" about those video.

I really appreciate your kind ffmpeg command!! I tried the command, but it also doesn't play fluently and doesn't use the "hardware Video Decoding". Therefore, I guess my laptop environment has problem.

I tried 2 videos as following:

---

1) test 1 This video is as same as the one I said before.

1-1) video information before converting

1-2) converting with ffmpeg

1-3) video information after converting

1-4) play

---

2) test 2 This AVI(hap) video is made by Davinci Resolve.

2-1) video information before converting

2-2) converting with ffmpeg

2-3) video information after converting

2-4) play

---

I already installed the Microsoft HEVC codec you suggested and I also installed the lav filter that Renderheads suggested in a document. Does my encoding setting or playing setting has problem...?

[AndrewRH]

Wait why do you have the Video API set to DirectShow? You should set it to Media Foundation. DirectShow has a lot of extra overhead.

Or if you have to use LAV Filters (DirectShow), then run LAV Video Configuration and select the H/W decoder to use NVIDIA CUVID:

I would also recommend going into the NVidia Control Panel and making sure the NVidia is set as the default GPU instead of Auto Detection.

Thanks,

[keitasumiya]

As written in the issue https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/86 , the reason is as followings:

First of all, I tried to play my 8K color4:4:4 video on windows UnityEditor via AVPro but it didn't work and a white board was just displayed. Therefore, I installed LAVFilters to my windows PC. After installing LAVFilters, 8K color4:4:4 video was displayed in windows UnityEditor via AVPro but it's not fluent.

Secondly, AVPro sometimes recommends using LAV filter in an official document: http://downloads.renderheads.com/docs/UnityAVProVideo.pdf . It's 2nd reason.

Lastly, Directshow option was effective for some specific AVI(hap) videos and I could play those some specific AVI(hap) videos fluently at that time. It is the 3rd reason why I still use LAVFilter(Directshow) until now.

However, I didn't use the "Hardware decoder to use" setting. Thanks to your advice, my player can't play fluently yet but looks like using NVidia Video Decording for some specific videos. Therefore I re-check all my test environment again from now!! Of course, it includes the re-test about the Media Foundation option you suggested. So would you wait little longer to test it? I am so sorry.

[keitasumiya]

I tested Media Foundation, LAV Filter (HW Decoding setting) for an video made by ffmpeg on 3 laptops. But, it could not play fluently...

1) video I made a test video by ffmpeg from tif sequence and the converting code is like (using code you suggested):

ffmpeg -hide_banner -y -r 60 -i tif\test_%05d.tif -pix_fmt yuv420p -r 60 -vf scale=7680x4320,setsar=1:1 -c:v libx265 -vtag hvc1 -tune fastdecode test_hevc-8k60.mp4

The result:

2) play the video on some laptops 2-1) Thinkpad P1 (Quadro P2000 4GB) 2-1-DS) Directshow I could play it but the display rate changes frequently from 0 to about 20 in a second (unstable).

2-1-MF) Media Foundation I could play it but the display rate is about 15 (stable).

2-2) Thinkpad (Quadro P5000 16GB) 2-2-DS) Directshow I could play it but the display rate changes frequently from 0 to about 20 in a second (unstable).

2-2-MF) Media Foundation I could play it but the display rate is about 18 (stable).

2-3) G-GEAR (Geforce GTX1070 8GB) 2-3-DS) Directshow I could play it but the display rate changes frequently from 0 to about 30 in a second (unstable).

2-3-MF) Media Foundation I could play it but the display rate is about 30 (stable).

Therefore, these display rate (about 20) is less than the fps of the video (60). You tell me that you already checked playing 8K60 video on GTX1050 etc. I think above my video cards is more powerful than GTX1050. So I don't know bad points of my environments. If you understand my bad points, please let me know it... I am really really sorry I caused you so much trouble yet.

[AndrewRH]

wow...very detailed testing!! I will look more later - but for you, could you try to disable the option "Use Low Latency" as this will decrease the decoder performance. Thanks,

[keitasumiya]

Thank you for your relpy!!

I tested enable/disable "Use Low Latency" on Thinkpad p1 (Quadro P2000 4GB). 1-MF) enable "Use Low Latency" I could play it but the display rate is about 10 (stable). This is as same environment as previous my coment (2-1-MF) Media Foundation). It's a little slower than before...?

2-MF) disable "Use Low Latency" I could play it but the display rate is about 10 (stable). It is almost the same as the above 1-MF) enable "Use Low Latency".

[AndrewRH]

Hmm..that's strange...I would have expected better performance. I see in your screenshot the CPU is only running at 1GHz which is strange. Please check: Is the laptop set up with power connected?
Is it set with the power-profile set to Maximum Performance (in Power settings Control Panel)?

[keitasumiya]

I am so sorry... As you said, My previous comment : https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/152#issuecomment-603147624 is the result which the laptop is NOT set up with power connected. (In more previous comment: https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/152#issuecomment-602297749 is the result which laptops are set up with power connected. )

Therefore, I re-test my previous one : https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/152#issuecomment-603147624 .

---

0-1) I set up my laptop (thinkpad p1) with power connected:

0-2) I checked my power setting as high performance:

1-1-MF) enable "Use Low Latency" I could play it but the display rate is about 15 (stable).

1-2-MF) disable "Use Low Latency" I could play it but the display rate is about 17 (stable).

As a result, disable setting (fps17) is a little bit faster than enable setting (fps15).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.