How to enable experimental hardware encoding

[kspeakman]

Question: The main readme on this github mentions the following under Features.

Experimental hardware encoding and decoding (VA-API, NVDEC, D3D9, D3D11, VTB)

How do I enable this experimental feature? Especially for encoding.

So far, I managed to have some limited success during exporting by going to Advanced and changing the Video Codec to nvenc_h264 or h264_nvenc. When I do this I can observe a reduced CPU usage and very slight video encoder usage on the GPU.

When exporting with the default libx264, CPU stays capped out and there is no Video Encode usage at all. It renders about 12-14 fps of 2.4k video.

However, the hardware encoder has a lot more head room left. Is there more I can do to utilize it? It still takes a long time to render (only rendering about 22 FPS 2.4k).

Thanks!

System Details:

• OpenShot Version: 2.4.4
• Operating System: Windows 10

[ferdnyc]

@kspeakman

IF hardware decoding or encoding are available on your system — I'm not sure they're actually functioning yet, on Windows and MacOS — then they'd be enabled in the "Performance" section of the Preferences.

Assuming you're running one of the recent Daily Builds, you should find a whole new set of hardware options there, and if your graphics card is detected you can select it to be used as the hardware decoder and/or encoder. If that selection is disabled, then that's probably a sign that my memory is correct, and the hardware features still aren't completed on Windows systems.

[ferdnyc]

(Oh, and if you are actually running OpenShot 2.4.4, then you won't find any of that. The hardware features are so experimental, they were only added in the past few months since the 2.4.4 release!)

[kspeakman]

@ferdnyc Wow, so this is interesting. I am not running a daily build, but rather the release version on the website. But when I go to encode and change the video codec, I do get GPU video encoder activity, reduced CPU load, and an encoding speedup: libx264 = ~2.75 hrs to render (estimated), nvenc_h264 = ~1.5 hrs, 20 min video at 1440p 96fps.

From lurking around here, I gather that libopenshot uses ffmpeg libraries. Is it possible that the codec option is passed into them, and they are making use of it to get some GPU acceleration?

[ferdnyc]

@kspeakman Absolutely, yeah, NVENC is Nvidia's hardware-accelerated encoder, and if ffmpeg is making it available to you then using it should take advantage of that.

The next release of OpenShot will be aware of that capability and be able to take even better advantage of that by specifically targeting the accelerated pipelines detected on the system. (Hopefully, on Windows, as it already does on Linux.) But I'm not surprised FFmpeg is making some of that available all on its own, now that we're compiling it with support for hardware accelerated codecs. (That started since either 2.4.4 or 2.4.3, I can't recall.)

[ferdnyc]

(As a teaser, here's what the Performance Preferences and the Encode window look like, in my current dev build:)

[kspeakman]

Oh my gosh, that looks awesome. I love it.

Rendering time has been a sticking point, so I can't wait for this!

As a side note, hopefully your detection code isn't similar to ShotCut's, because it cannot detect my video card. Kdenlive also couldn't find it. But OBS does find/use it. It is a little bit of an odd duck: Gigabyte G1 Gaming 980 Ti. Gigabyte used a custom PCB to put 2 DVI connectors on it, and I got it specifically to use with DVI-based overclockable monitors. Still the same Nvidia chips, so it shouldn't matter, but it makes me insecure about compatibility sometimes. I digress...

Thanks so much for your answers and the work you do!

[ig0rnig0r]

Hi there,

guys you really do a great job in going forward with the GPU support! Many of us really appreciate your work!

Since I'm desperately looking for a video editor compatible with my quite uncommon setup (x230 + expresscard Nvidia GTX970 eGPU) I came across the awesome new features and installed the 2.5.0 dev version. While I do already see the new performance options, I miss the (for Nvidia users crucial) setting about "VDPAU". It seems that openshot recognises the internal Intel "GPU", but not the external Nvidia one.

At least in the interesting article about the experimental libopenshot HARDWARE_DECODER setting it sais something about "VDPAU for some reason needs a card number one higher than it really is", so maybe this is related. I'm pretty sure that all drivers are installed correctly, since "mpv --hwdec=vdpau test.mp4" tells me that indeed he's using the GPU (also visible in the nvidia-smi).

In any case I'd gladly offer my setup in case you guys want to do some testing or debugging and would really look forward to any news on that topic. Some specific hints on my setup would be super helpful!

Thanks so much, and keep up the superb work!! Best regards, Christopher

[ferdnyc]

@ig0rnig0r Sorry for the delay in getting back to you.

Hmm... so you have a switchable-GPU system, the kind that normally uses an Intel chip, but has the Nvidia chip on hand for on-demand performance boosts? Optimus, I think the tech is called? I'm aware of those setups, though personally I've never worked with one. My Nvidia card is the primary GPU for my system.

A couple of questions that may help nail this down:

• What nvidia devices are present on your system, if any? ls /dev/nvidia*
• What DRI devices? ls /dev/dri/*
• I assume you still need drivers for the Nvidia side of things, as well? Are you using Nouveau, or the Nvidia binary drivers? What driver version?
• Are you using X.org or Wayland, as your display server?
• If X.org, what's the output of xrandr --listproviders?
• Do you use (or have available) a tool like primusrun/prime-run/prime-select to switch into or launch specific applications in Nvidia PRIME mode? If you run OpenShot using those tools, does anything change?

[ferdnyc]

Oh, yes, two more things:

• What's the output of vdpauinfo and vainfo, on your machine?
• Does the h.264 nv (NVENC) and/or h.264 va (VA-API) encoder show up in the Encode window's Target selector? If you select either of those, can you successfully encode a video?

[ig0rnig0r]

@ferdnyc, first of all thanks so much for comming back on my question - really great!

Sorry for the late reply, busy life.

Before I'll try to answer all points as good as possible, I'd like to clearify that I do not use optimus in my system. I tried it at first within debian a while ago, but could not get any performance increase anywhere starting dedicated jobs (e.g. vlc, mpv, kdenlive, openshot, etc) via the opti-run (bumblebee) stuff. Maybe the problem was that I did not even get the secondary monitor running via the eGPU (which btw is really standing beside my notebook, connected via the expresscard slot). Instead, everything was going back and forth between the expresscard interface and the monitor was connected to the VGA port on my docking station.

The solution was to install ubuntu in UEFI mode (instead of standard bios) which apparently kind of bypassed some hardware checks for the x230 to actually boot with an external monitor connected to the eGPU (via HDMI). Long story short, using ubuntu 18 almost everything worked out of the box (wow folks..great stuff!!). I followed this instructions to install my nvidia drivers (through the ubuntu repository): https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-18-04-bionic-beaver-linux

From there it took some experimenting, but at least I got some major performance improvements while playing some GoPro footage, and you could tell from the system monitor that actually the eGPU is doing all the heavy lifting (compared to the CPU before..).

• What nvidia devices are present on your system, if any? ls /dev/nvidia* /dev/nvidia0 /dev/nvidiactl /dev/nvidia-modeset /dev/nvidia-uvm /dev/nvidia-uvm-tools

• What DRI devices? ls /dev/dri/* /dev/dri/by-path: pci-0000:00:02.0-card pci-0000:00:02.0-render pci-0000:04:00.0-card pci-0000:04:00.0-render

• I assume you still need drivers for the Nvidia side of things, as well? Are you using Nouveau, or the Nvidia binary drivers? What driver version? As mentioned above, I'm pretty confident that the nvidia drivers are installed properly. Here's a screenshot of my nvidia settings: Also, entering lsmod | grep nouveau does not return anything, so I think I don't use the nouveau drivers. According to nvidia-smi the driver version is 435.21 (second screenshot from my last post). Also, software cencter shows me this:

• Are you using X.org or Wayland, as your display server? It seems to be wayland, but I also have a second session. Check this out:

• If X.org, what's the output of xrandr --listproviders? So somehow I have both running, X.org and wayland? Output:

• Do you use (or have available) a tool like primusrun/prime-run/prime-select to switch into or launch specific applications in Nvidia PRIME mode? If you run OpenShot using those tools, does anything change? It seems I do have prime-select and prime-switch installed, but also it looks like the system already uses the GTX970 as a primary GPU:

Looks like it is already a part of the whole nvidia driver, since PRIME profiles show up in the nvidia settings already:

So I think the whole system uses the external GPU already..

• What's the output of vdpauinfo and vainfo, on your machine? ig0r@ig0r-ThinkPad-X230:~$ vdpauinfo display: :0 screen: 0 API version: 1

Information string: NVIDIA VDPAU Driver Shared Library 435.21 Sun Aug 25 08:06:02 CDT 2019

Video surface:

name width height types

420 4096 4096 NV12 YV12 422 4096 4096 UYVY YUYV 444 4096 4096

Decoder capabilities:

name level macbs width height

MPEG1 0 65536 4080 4080 MPEG2_SIMPLE 3 65536 4080 4080 MPEG2_MAIN 3 65536 4080 4080 H264_BASELINE 51 65536 4096 4096 H264_MAIN 51 65536 4096 4096 H264_HIGH 51 65536 4096 4096 VC1_SIMPLE 1 8190 2048 2048 VC1_MAIN 2 8190 2048 2048 VC1_ADVANCED 4 8190 2048 2048 MPEG4_PART2_SP 3 8192 2048 2048 MPEG4_PART2_ASP 5 8192 2048 2048 DIVX4_QMOBILE 0 8192 2048 2048 DIVX4_MOBILE 0 8192 2048 2048 DIVX4_HOME_THEATER 0 8192 2048 2048 DIVX4_HD_1080P 0 8192 2048 2048 DIVX5_QMOBILE 0 8192 2048 2048 DIVX5_MOBILE 0 8192 2048 2048 DIVX5_HOME_THEATER 0 8192 2048 2048 DIVX5_HD_1080P 0 8192 2048 2048 H264_CONSTRAINED_BASELINE 51 65536 4096 4096 H264_EXTENDED 51 65536 4096 4096 H264_PROGRESSIVE_HIGH 51 65536 4096 4096 H264_CONSTRAINED_HIGH 51 65536 4096 4096 H264_HIGH_444_PREDICTIVE 51 65536 4096 4096 HEVC_MAIN --- not supported --- HEVC_MAIN_10 --- not supported --- HEVC_MAIN_STILL --- not supported --- HEVC_MAIN_12 --- not supported --- HEVC_MAIN_444 --- not supported ---

Output surface:

name width height nat types

B8G8R8A8 16384 16384 y Y8U8V8A8 V8U8Y8A8 A4I4 I4A4 A8I8 I8A8 R10G10B10A2 16384 16384 y Y8U8V8A8 V8U8Y8A8 A4I4 I4A4 A8I8 I8A8

Bitmap surface:

name width height

B8G8R8A8 16384 16384 R8G8B8A8 16384 16384 R10G10B10A2 16384 16384 B10G10R10A2 16384 16384 A8 16384 16384

Video mixer:

feature name sup

DEINTERLACE_TEMPORAL y DEINTERLACE_TEMPORAL_SPATIAL y INVERSE_TELECINE y NOISE_REDUCTION y SHARPNESS y LUMA_KEY y HIGH QUALITY SCALING - L1 y HIGH QUALITY SCALING - L2 - HIGH QUALITY SCALING - L3 - HIGH QUALITY SCALING - L4 - HIGH QUALITY SCALING - L5 - HIGH QUALITY SCALING - L6 - HIGH QUALITY SCALING - L7 - HIGH QUALITY SCALING - L8 - HIGH QUALITY SCALING - L9 -

parameter name sup min max

VIDEO_SURFACE_WIDTH y 1 4096 VIDEO_SURFACE_HEIGHT y 1 4096 CHROMA_TYPE y
LAYERS y 0 4

attribute name sup min max

BACKGROUND_COLOR y
CSC_MATRIX y
NOISE_REDUCTION_LEVEL y 0.00 1.00 SHARPNESS_LEVEL y -1.00 1.00 LUMA_KEY_MIN_LUMA y
LUMA_KEY_MAX_LUMA y

• Does the h.264 nv (NVENC) and/or h.264 va (VA-API) encoder show up in the Encode window's Target selector? If you select either of those, can you successfully encode a video? As a matter of fact, both of them show up, and yes, both of them can export a video successfully. I did a quick test for a 1 minute video and both options took ~2:15 minutes. The CPU seemed to take the same loading, however, in case of the NVENC option, the GPU temperature was significantly higher (~56°C compared to the VA-API at ~49°C, leaving the laptop alone while encoding). Seems that the GPU is indeed loaded, but there are no performance gains and same CPU loading?

Thanks again for asking these very interesting questions, maybe other people will find some clues to debug their settings as well here. As always, I'm happy to help with some testing in this area - and any inputs from your side are highly appreciated, of course! Still looking forward to get this thing running properly!

All the best to you guys, cheers!

[ig0rnig0r]

Ah, a quick note concerning the encoder settings question:

While none of the possible encoder/decoder selections within the performance tab in the OpenShot preferences show anything about the nvidia GPU (no NVENC, just CPU and VA-API), the export video pop-up does show NVENC as a possible target (this is how I run my test exports).

I thought this might be an useful info too. Thanks! Keep crushing!

[ig0rnig0r]

@ferdnyc, is there any chance we can follow up on this?

I saw that Jonathan officially posted the (great!!) news on hardware support here: https://www.openshot.org/blog/2020/02/08/openshot-250-released-video-editing-hardware-acceleration/

The Documentation on of libopenshot seems to be updated too (http://openshot.org/files/libopenshot/md_doc_HW-ACCEL.html) and still makes me believe that I should be good to go with my existing setup. By now, I think these settings work out of the box, or should I enable those features somehow manually?

BTW, I recently also played some CS:GO via Steam on this ubuntu lenovo x230, and the graphics performances was awesome, since the eGPU is fully utilized.

However, even though I run openshot 2.5.1-dev2 by now, the second graphics card is still not recognised :( According to the questions and answers from the last posts, did you find any clues on why this might fail for me? Or everything is set up properly?

Any info on this topic would be super great. I tried to involve Jonathan also a bit but no luck so far. Thanks so much! All the best and health! Christopher

[ferdnyc]

@ig0rnig0r Ah, OK, that answers the question I'd just rhetorically asked in #3251 about whether or not preferences GPU detection was working yet in Windows, thanks! (The answer, not unexpectedly, seems to be no. Or at least, not yet.)

However, as you say, if the accelerated encoders show up in the Export dialog list, then they should be usable at least on your system's primary GPU, when exporting in a supported format. And that is, indeed, the lion's share of hwaccel's usefulness ­— even though we present the option if we're capable of activating it, hardware-accelerated decoding is of precious little benefit to OpenShot, as I think we've discussed in the past.

So, by all means — if there are any green ovals in the export Target list that match your hardware, make use of them! They should offer a nice little speedup. I just ran a quick test on my own Linux system, and even with my woefully underwhelming Nvidia GT 710, the export time for a simple, one-video Timeline (encoding to 1280×720@30 fps, High quality, in both cases) went from 35 seconds / 33 fps with the software encoder, to 17 seconds / ~65fps using NVENC MP4. Users with halfway-decent cards should see even bigger gains.

[ig0rnig0r]

@ferdnyc, thanks for your message!

I'm not quite sure how I answered your question concerning a Microsoft windows topic while posting about my ubuntu issues :D :D Anyway, yes you are right I may use NVENC as the export target (which works, but didn't show any encode speed improvements so far, as described in the post from 15.02.2020 here).

The real issue however is that the preview playback of a h264 GoPro Hero 7 footage video in OpenShot is extremely slow and therefore the whole editor as such is practically not usable (as all the others I tried so far). I thought this is due to that fact that I can not switch to the eGPU within the hardware decoder performance settings (as described), since I see fully CPU load. Interestingly you just mentioned that the hardware-accelerated decoding is of little benefit in OpenShot, "as we've discussed in the past". Could you please point me to this discussion? This would be a real bummer, as I thought this exact setting is basically giving me a playback speed improvement similar seen in VLC and mpv player under ubuntu with my eGPU setup (from having 100% CPU usage and still lagy video playback to almost no CPU utilisation and a flawless video, since the eGPU is loaded and uses their hardware decoders).

Please lets follow up on this a bit more, since I still think your great editor has full potential on my weird setup.

Thanks and all the best! Christopher

[ig0rnig0r]

@ferdnyc, I just saw that I never answered one of your questions:

• What's the output of vdpauinfo and vainfo, on your machine? ig0r@ig0r-ThinkPad-X230:~$ vainfo libva info: VA-API version 1.1.0 libva info: va_getDriverName() returns 0 libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so libva info: Found init function __vaDriverInit_1_0 libva info: va_openDriver() returns 0 vainfo: VA-API version: 1.1 (libva 2.1.0) vainfo: Driver version: Splitted-Desktop Systems VDPAU backend for VA-API - 0.7.4 vainfo: Supported profile and entrypoints VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointVLD VAProfileMPEG4Simple : VAEntrypointVLD VAProfileMPEG4AdvancedSimple : VAEntrypointVLD : VAEntrypointVLD VAProfileH264Main : VAEntrypointVLD VAProfileH264High : VAEntrypointVLD VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD

Unfortunately, I'm still completely lost here. Since I also try out different free editors, according to cinelerra-gg (which is the second best hope for me) it seems to be an issue with ffmpeg. So in case anybody has a clue on how to debug the ffstream / ffmpeg stuff please let me know. This is what I'll do next.

Any inputs are highly appreciated! Thanks! Best regards, Christopher

[phylsmith2004]

@ferdnyc You were helpful to us in the past so I wanted to pass this along in case it is helpful for OpenShot as it was for a different NLE. There is an ffmpeg patch that when included in this other NLE, allowed for vdpau to handle yuvj420p where as before it would not. The patch is listed at: https://github.com/mpv-player/mpv/issues/6122

[MJanich]

Hi,

I've the same issue with my RX580. When I open the setting dialog from the appimage v2.5.1 started from the console I get these lines here. Maybe it helps in finding the problem:

ui_util:INFO Initializing UI for dlgPreferences metrics:INFO Track metric: [200] http://www.google-analytics.com/collect?v=1&cid=05438029-1851-460d-b5db-ff0bf140456c&aid=org.openshot.openshot-qt&cd2=3.4.3&t=screenview&tid=UA-4381101-5&cd5=Ubuntu-18.04-bionic&ua=Mozilla%2F5.0+%28X11%3B+Linux+x86_64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F37.0.2062.120+Safari%2F537.36&an=OpenShot+Video+Editor&cd=preferences-screen&av=2.5.1&ul=en-us&aip=1&cd3=5.2.1&cd4=5.2.1&cd1=0.2.5 | (35 bytes) Hardware decoding device number: 0 libva info: VA-API version 0.39.4 libva info: va_getDriverName() returns -1 libva error: va_getDriverName() failed with unknown libva error,driver_name=(null) [AVHWDeviceContext @ 0x2dfec20] Failed to initialise VAAPI connection: -1 (unknown libva error). Hardware decoding device number: 0 libva info: VA-API version 0.39.4 libva info: va_getDriverName() returns -1 libva error: va_getDriverName() failed with unknown libva error,driver_name=(null) [AVHWDeviceContext @ 0x2ee0220] Failed to initialise VAAPI connection: -1 (unknown libva error). preferences:WARNING Exception trying to test hardware decoding in preferences (this is expected): 1-0 Hardware decoding device number: 1 libva info: VA-API version 0.39.4 libva info: va_getDriverName() returns 0 libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so libva error: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function vaDriverInit_0_32 libva info: va_openDriver() returns -1 [AVHWDeviceContext @ 0x2ee3380] Failed to initialise VAAPI connection: -1 (unknown libva error). Hardware decoding device number: 1 libva info: VA-API version 0.39.4 libva info: va_getDriverName() returns 0 libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so libva error: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function vaDriverInit_0_32 libva info: va_openDriver() returns -1 [AVHWDeviceContext @ 0x31d7120] Failed to initialise VAAPI connection: -1 (unknown libva error). preferences:WARNING Exception trying to test hardware decoding in preferences (this is expected): 1-1 Hardware decoding device number: 0

running on Kubuntu 18.04.

Could it be that there is some kind of version mismatch since vainfo also says libva info: VA-API version 1.1.0

but above it is version 0.39.4 printed?

UPDATE: Building the libs on my own and launching the the application with the latest version from Github as described here https://github.com/OpenShot/openshot-qt/wiki/Become-a-Developer solved the issue. So for me it looks like an issue with the appimage.

[Yrij-Zhavoronkov]

Win 8.1 FFmpeg + h264_nvenc take 200+ Fps, but openshot 2.5.1 can't use hardware acs, why?

[zaelu]

Same as above for me. I have a GTX1070 (GP104) and I can encode decode in OBS but not in OpenShot. https://en.wikipedia.org/wiki/Nvidia_NVENC

Any clues? Thanks!

[Yrij-Zhavoronkov]

There are assumptions. The project is raw that's what. Apart from the lack of hardware encoding there are many other bugs:

• shift the audio forward or backward in some of the video (e.g. from Huawei phones)
• when cropping a video, one thing is visible in the preview. the output is different.
• small number of effects
• inability to set 100+% audio volume (ffmpeg supports this) , etc...

[AGPX]

I've OpenShot 3.2.1 in my system (Windows 11 + Nvidia RTX 4080), but the NVENC is not available. Hardware encoding is very important because encoding time is very long (I've also a 16 core processor, but looks like it uses only 1 core during exporting!)

[Colorjet3]

Hello @AGPX Hardware acceleration is not well supported at this time since v2.6.x and not much progress has been made. I also don't have an ETA on when it will be worked on.