Hardware acceleration: i915kms.ko: Operation not permitted

[jssilva25]

Hello, Trying to install hardware acceleration following the instructions; when reaching:

/root/enable_gpu_jails.sh
devfs rule: ioctl DEVFSIO_RGETNEXT: Operation not permitted
/etc/rc.d/devfs: WARNING: devfs_init_rulesets: could not read rules from /etc/defaults/devfs.rules
kldload: can't load /boot/modules/i915kms.ko: Operation not permitted

This is of course before changing devfs from 4 to 10 on the jail properties.

But strangely my TrueNas seems to be working with Intel QSV; or else I don't understand the subject. My CPU is Intel(R) Xeon(R) CPU E3-1286 v3 (4th Gen)

Thank you for this port.

[Thefrank]

That script should be executed on the host (TrueNAS's shell) and not in the jail itself.

I double checked the script. It correctly appends the two new rules, restarts the service, and loads the kernel module...even on a system with no supported graphics.

The server market is odd.

Your CPU does support iQSV: https://www.intel.com/content/www/us/en/products/sku/80913/intel-xeon-processor-e31286-v3-8m-cache-3-70-ghz/specifications.html

Mine is too old: https://www.intel.com/content/www/us/en/products/sku/64584/intel-xeon-processor-e52660-20m-cache-2-20-ghz-8-00-gts-intel-qpi/specifications.html

[jssilva25]

Thank you for replying.

How silly of me. I didn't interpret it that way. Yes, the script does not show an error now. So, I'm past that now.

But I get only errors with vainfo:

Trying display: wayland
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
error: failed to initialize display

Btw, I think my cpu is 4th gen, so I installed libva-intel-driver.

Yes, I think I made a good deal buying my used SuperMicro board with that cpu plus 32 GiB ECC ram.

The best thing is that it only consumes 25 Watt idle with 2 x 14TB + 2 x 3TB WDRed disks.

Thank you again for your time.

[jssilva25]

Just a parallel information, probably useful to people reading this site, although perhaps not the best place to post it:

I have successfully implemented a TvHeadEnd 4.3 server on a VM with Debian 11 running on TrueNas, which feeds the TvHeadEnd plugin of the Jellyfin server in a jail of the same server. Everything is working: TV channels, recordings, EPG, ...

I plugged two USB DVB sticks onto a PCIe x1 with 4 USB ports, which I could successfully pass through to the TVHeadEnd VM. This solution doesn't force me install things on the base OS, like I would have to do if running TVH in a jail, which is very unadvisable by the TrueNAS gurus.

This is a better solution than I had before, an Rpi 3B+ running the TVHeadEnd on Debian. It consumes less power and is much less cumbersome.

Can provide details if someone is interested.

[Thefrank]

That driver should work. Your device is here (P4700): https://en.wikipedia.org/wiki/Intel_Graphics_Technology#Gen7.5_architecture

TrueNAS does not ship drm-kmod but this table agrees on what driver you should be using: https://forums.freebsd.org/threads/drm-kmod-help-completing-the-compatibility-matrix-intel-gpus-vs-freebsd-versions.79362/

Are /dev/dri* or /dev/drm* showing up in the jail?

[jssilva25]

ls /dev/dri*
ls: No match.

ls /dev/drm*
0   121 145 169 192 215 239 32  56  8
1   122 146 17  193 216 24  33  57  80
10  123 147 170 194 217 240 34  58  81
100 124 148 171 195 218 241 35  59  82
101 125 149 172 196 219 242 36  6   83
102 126 15  173 197 22  243 37  60  84
103 127 150 174 198 220 244 38  61  85
104 128 151 175 199 221 245 39  62  86
105 129 152 176 2   222 246 4   63  87
106 13  153 177 20  223 247 40  64  88
107 130 154 178 200 224 248 41  65  89
108 131 155 179 201 225 249 42  66  9
109 132 156 18  202 226 25  43  67  90
11  133 157 180 203 227 250 44  68  91
110 134 158 181 204 228 251 45  69  92
111 135 159 182 205 229 252 46  7   93
112 136 16  183 206 23  253 47  70  94
113 137 160 184 207 230 254 48  71  95
114 138 161 185 208 231 255 49  72  96
115 139 162 186 209 232 26  5   73  97
116 14  163 187 21  233 27  50  74  98
117 140 164 188 210 234 28  51  75  99
118 141 165 189 211 235 29  52  76
119 142 166 19  212 236 3   53  77
12  143 167 190 213 237 30  54  78
120 144 168 191 214 238 31  55  79

[Thefrank]

Is the jail using the correct devfs ruleset? does the output from inside the jail match the host? does pciconf -lvbce list your device (should have the whole name)? does devinfo -vr list it too (this will be something a little as vgapcixxx)? This typically matches the same name in dmesg does sysctl -d hw.i915kms output list of options/overrides?

[jssilva25]

I must confess I'm having a bit difficulty to follow you. I hope this is what you asked for.

yes:

I wonder what you mean, but I get this on the host:

root@freenas:~ # devfs rule showsets
1
2
3
4
5
10
100
101
1000
1001
1002
1003

10 is shown

But, on the jail

root@jellyfin-server:/ # devfs rule showsets
devfs rule: ioctl DEVFSIO_SGETNEXT: Operation not permitted

On the host:

root@freenas:~ # pciconf -lvbce | grep P4700

gets nothing

root@freenas:~ # pciconf -lvbce
...
vgapci0@pci0:4:0:0: class=0x030000 rev=0x30 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x15d9 subdevice=0x0803
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
    bar   [10] = type Memory, range 32, base 0xf6000000, size 16777216, enabled
    bar   [14] = type Memory, range 32, base 0xf7000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0xe000, size 128, enabled
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 4 messages, 64 bit 
...

This is the only graphics device I can find

On the jail:

root@jellyfin-server:/ # pciconf -lvbce
pciconf: /dev/pci: No such file or directory

On the host:

root@freenas:~ # devinfo -r
...
  vtvga0
      I/O memory addresses:
          0xa0000-0xaffff
...

Same on the jail.

On the host, and I think it's the same on the jail, long list:

root@freenas:~ # sysctl -d hw.i915kms
hw.i915kms: Intel Graphics parameters
hw.i915kms.enable_dpcd_backlight: Enable support for DPCD backlight control(-1=use per-VBT LFP backlight type setting, 0=disabled [default], 1=enabled)
hw.i915kms.enable_dp_mst: Enable multi-stream transport (MST) for new DisplayPort sinks. (default: true)
hw.i915kms.guc_log_level: GuC firmware logging level. Requires GuC to be loaded. (-1=auto [default], 0=disable, 1..4=enable with verbosity min..max)
hw.i915kms.enable_guc: Enable GuC load for GuC submission and/or HuC load. Required functionality can be selected using bitmask values. (-1=auto, 0=disable [default], 1=GuC submission, 2=HuC load)
hw.i915kms.edp_vswing: Ignore/Override vswing pre-emph table selection from VBT (0=use value from vbt [default], 1=low power swing(200mV),2=default swing(400mV))
hw.i915kms.nuclear_pageflip: Force enable atomic functionality on platforms that don't have full support yet.
hw.i915kms.verbose_state_checks: Enable verbose logs (ie. WARN_ON()) in case of unexpected hw state conditions.
hw.i915kms.mmio_debug: Enable the MMIO debug code for the first N failures (default: off). This may negatively affect performance.
hw.i915kms.disable_display: Disable display (default: false)
hw.i915kms.invert_brightness: Invert backlight brightness (-1 force normal, 0 machine defaults, 1 force inversion), please report PCI device ID, subsystem vendor and subsystem device ID to dri-devel@lists.freedesktop.org, if your machine needs it. It will then be included in an upcoming module version.
hw.i915kms.force_reset_modeset_test: Force a modeset during gpu reset for testing (default:false). For developers only.
hw.i915kms.load_detect_test: Force-enable the VGA load detect code for testing (default:false). For developers only.
hw.i915kms.prefault_disable: Disable page prefaulting for pread/pwrite/reloc (default:false). For developers only.
hw.i915kms.fastboot: Try to skip unnecessary mode sets at boot time (0=disabled, 1=enabled) Default: -1 (use per-chip default)
hw.i915kms.enable_ips: Enable IPS (default: true)
hw.i915kms.disable_power_well: Disable display power wells when possible (-1=auto [default], 0=power wells always on, 1=power wells disabled when possible)
hw.i915kms.alpha_support: Deprecated. See i915.force_probe.
hw.i915kms.enable_psr: Enable PSR (0=disabled, 1=enabled) Default: -1 (use per-chip default)
hw.i915kms.enable_hangcheck: Periodically check GPU activity for detecting hangs. WARNING: Disabling this can cause system wide hangs. (default: true)
hw.i915kms.error_capture: Record the GPU state following a hang. This information in /sys/class/drm/card<N>/error is vital for triaging and debugging hangs.
hw.i915kms.reset: Attempt GPU resets (0=disabled, 1=full gpu reset, 2=engine reset [default])
hw.i915kms.vbt_sdvo_panel_type: Override/Ignore selection of SDVO panel mode in the VBT (-2=ignore, -1=auto [default], index in VBT BIOS table)
hw.i915kms.panel_use_ssc: Use Spread Spectrum Clock with panels [LVDS/eDP] (default: auto from VBT)
hw.i915kms.lvds_channel_mode: Specify LVDS channel mode (0=probe BIOS [default], 1=single-channel, 2=dual-channel)
hw.i915kms.enable_fbc: Enable frame buffer compression for power savings (default: -1 (use per-chip default))
hw.i915kms.enable_dc: Enable power-saving display C-states. (-1=auto [default]; 0=disable; 1=up to DC5; 2=up to DC6)
hw.i915kms.modeset: Use kernel modesetting [KMS] (0=disable, 1=on, -1=force vga console preference [default])

Sorry if I was not concise.

[Thefrank]

It looks like the OS is only detecting the BMC's (Supermicro's) onboard graphics. Please go into the BIOS/UEFI and make sure that iGPU is turned on and has enough RAM assigned to it

Github allows you to put multiple lines of code in to code block if you use triple "`" at the line above and below the block:

like
this

It makes it easier to read :)

[jssilva25]

Very clever. I probably disabled it long time ago (who needs a graphics display on a server?) and completely forgot.

Now I have (with the block display, thank you):

root@jellyfin-server:/ # vainfo
Trying display: wayland
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: va_openDriver() returns -1
libva info: Trying to open /usr/local/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.17.1)
vainfo: Driver version: Intel i965 driver for Intel(R) Haswell Server - 2.4.1
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264MultiviewHigh      : VAEntrypointVLD
      VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileH264StereoHigh         : VAEntrypointEncSlice
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD

But, after changing QSV to VAAPI it's not working, gives me a black screen after the revolving circle; no message incompatible format, bla, bla, ... at least formats .ts and .avi; but works with mp4.

If I revert to QSV hardware acceleration and reinstate ffmpeg, everything works. So, after using so much of your time this might seem a silly question: why do I need VAAPI at all if I have QSV? What do I gain from VAAPI?

[Thefrank]

What I have found in past experiences with desktop usage:

Image quality is typically better with VAAPI but QSV is sometimes faster and in a few cases has better decode support. QSV requires you to install this monster: https://www.freshports.org/multimedia/intel-media-sdk/ which only lists support for gen8 graphics (Broadwell) and later

So: VAAPI: better image quality, more platforms are supported QSV: faster in some cases, better decode on newer chips

[jssilva25]

I'm really learning a lot from you, thank you.

You say I need the sdk monster but I don't think I have it and everything is working as much as I can tell. How can this be?

Getting back to VAAPI, now that we are close to make it fully working, tomorrow I'll make some tests to check what is and what isn't working. Do you advise a particular test method?

[jssilva25]

It's not easy to test. I've set it for QSV and used another device for playing while I watched the dashboard, click info and read whether it is transcoding. Then,, I don't have those many formats and muxes but I saw that it is in fact transcoding some formats, for instance MPEG2.

What puzzles me is how it is transcoding considering I don't have, or at least don't remember to have installed, the monster Intel sdk. Can you explain this?

Then I set it for VAAPI and lffmpeg and did the same. The result is similar on common formats, for instance h264. But, for MPEG TS which is the format coming out of TVHeadEnd for a TV recording, the dashboard says it's being converted but it never starts. The playback time at the bottom goes until 0:10 and then rewinds to 0:00. The folder /var/db/jellyfinserver/transcodes is empty, if it means something.

With QSV, this format transcodes instantly. The dashboard info is:

Transcoding
The media is being converted into a format that is compatible with the device that is playing the media.
Framerate: 313fps

6.2 Mbps TS H264 AAC

Reason for transcoding:
The container is not supported
The video codec is not supported
The audio codec is not supported

On the other hand, although I didn't measure it yet, I wouldn't be surprised if the idle power consumption rises a bit by enabling the GPU.

So, all in all, on my side I'm happy with QSV and don't mind turning VAAPI off and stop using your time, although I would like to know how is it able to function without the sdk.

Would you care to comment?

[Thefrank]

the logs should be very clear. Here is an abstracted example:

[CODEC @ SOMETHINGHEX ] Using AUTO/VAAPI/INTEL/CUDA/SOMETHING hwaccel type VAAPI/QSV/VDAPU/SOMETHING with NAMEOFDEVICE device

if just grep'ing hwaccel type should be enough.

[jssilva25]

Using VAAPI: Played a movie requiring transcoding:

[2023-02-12 11:24:51.483 +00:00] [INF] [71] Jellyfin.Api.Helpers.MediaInfoHelper
: StreamBuilder.BuildVideoItem( Profile="Anonymous Profile", Path="/mnt/multimed
ia/Videos/externos/Filmes/A/5 Days Of War.avi", AudioStreamIndex=2, SubtitleStre
amIndex=0 ) => ( PlayMethod=Transcode, TranscodeReason=ContainerNotSupported, Vi
deoCodecNotSupported, AudioCodecNotSupported ) "media:/videos/841245c5-7c77-2546
-5300-eb6c3e20a61b/master.m3u8?MediaSourceId=841245c57c7725465300eb6c3e20a61b&Vi
deoCodec=h264&AudioCodec=aac,mp3&AudioStreamIndex=2&VideoBitrate=139616000&Audio
Bitrate=384000&MaxFramerate=23.976025&api_key=<token>&TranscodingMaxAudioChannel
s=2&RequireAvc=false&Tag=dade1becf9fd49c660f8d123f1ef41be&SegmentContainer=ts&Mi
nSegments=1&BreakOnNonKeyFrames=False&mpeg4-level=5&mpeg4-videobitdepth=8&mpeg4-
profile=advancedsimpleprofile&h264-profile=high,main,baseline,constrainedbaselin
e&h264-rangetype=SDR&h264-level=52&h264-deinterlace=true&TranscodeReasons=Contai
nerNotSupported,%20VideoCodecNotSupported,%20AudioCodecNotSupported"

Dashboard shows the play time clock going to 0:10 and cycling back to 0:00. Player shows the fanart of the movie and the clock at 0:00 But transcoding starts:

[2023-02-12 11:24:54.155 +00:00] [INF] [80] Jellyfin.Api.Helpers.TranscodingJobH
elper: "/usr/local/bin/lffmpeg" "-analyzeduration 200M  -i file:\"/mnt/multimedi
a/Videos/externos/Filmes/A/5 Days Of War.avi\" -map_metadata -1 -map_chapters -1
 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v
4499028 -maxrate 4499028 -bufsize 8998056 -profile:v:0 high -force_key_frames:0
\"expr:gte(t,0+n_forced*3)\" -vf \"setparams=color_primaries=bt709:color_trc=bt7
09:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(720\,304*a))/2)*2:trunc(m
in(max(iw/a\,ih)\,min(720/a\,304))/2)*2,format=nv12,hwupload=derive_device=vaapi
\" -codec:a:0 aac -ac 2 -ab 384000 -af \"volume=2\" -copyts -avoid_negative_ts d
isabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_s
egment_type mpegts -start_number 0 -hls_segment_filename \"/var/db/jellyfinserve
r/transcodes/c183298b82c80248ea8ab2745402c346%d.ts\" -hls_playlist_type vod -hls
_list_size 0 -y \"/var/db/jellyfinserver/transcodes/c183298b82c80248ea8ab2745402
c346.m3u8\""

Several:

[2023-02-12 11:25:02.629 +00:00] [WRN] [51] Jellyfin.Server.Middleware.ResponseT
imeMiddleware: Slow HTTP Response from "https://myserver/Sessions/Playing/Progress" to "192.168.1.2" in 0:00:00.6896627 with Status Code 204

Until cancelled by me, I guess:

[2023-02-12 11:26:39.453 +00:00] [ERR] [39] Jellyfin.Server.Middleware.Exception
Middleware: Error processing request: "A task was canceled". URL "GET" "/videos/
841245c5-7c77-2546-5300-eb6c3e20a61b/hls1/main/0.ts".

dvefs rule 10 is instanced. grep'd the whole jail for 'hwaccel'. The only result was before the play:

[2023-02-12 10:20:51.138 +00:00] [INF] [65] MediaBrowser.MediaEncoding.Encoder.M
ediaEncoder: Available hwaccel types: ["vdpau", "cuda", "vaapi", "drm"]

Is there any log option to enable showing hwaccel? I couldn't find it.

Using QSV:

[2023-02-12 11:40:54.339 +00:00] [INF] [15] Jellyfin.Api.Helpers.MediaInfoHelper
: StreamBuilder.BuildVideoItem( Profile="Anonymous Profile", Path="/mnt/multimed
ia/Videos/externos/Filmes/A/5 Days Of War.avi", AudioStreamIndex=2, SubtitleStre
amIndex=0 ) => ( PlayMethod=Transcode, TranscodeReason=ContainerNotSupported, Vi
deoCodecNotSupported, AudioCodecNotSupported ) "media:/videos/841245c5-7c77-2546
-5300-eb6c3e20a61b/master.m3u8?MediaSourceId=841245c57c7725465300eb6c3e20a61b&Vi
deoCodec=h264&AudioCodec=aac,mp3&AudioStreamIndex=2&VideoBitrate=139616000&Audio
Bitrate=384000&MaxFramerate=23.976025&api_key=<token>&TranscodingMaxAudioChannel
s=2&RequireAvc=false&Tag=dade1becf9fd49c660f8d123f1ef41be&SegmentContainer=ts&Mi
nSegments=1&BreakOnNonKeyFrames=False&mpeg4-level=5&mpeg4-videobitdepth=8&mpeg4-
profile=advancedsimpleprofile&h264-profile=high,main,baseline,constrainedbaselin
e&h264-rangetype=SDR&h264-level=52&h264-deinterlace=true&TranscodeReasons=Contai
nerNotSupported,%20VideoCodecNotSupported,%20AudioCodecNotSupported"

[2023-02-12 11:40:55.517 +00:00] [INF] [45] Jellyfin.Api.Helpers.TranscodingJobH
elper: "/usr/local/bin/ffmpeg" "-analyzeduration 200M  -i file:\"/mnt/multimedia
/Videos/externos/Filmes/A/5 Days Of War.avi\" -map_metadata -1 -map_chapters -1
-threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -preset ultrafast -crf
 23 -maxrate 4499028 -bufsize 8998056 -profile:v:0 high -level 51 -x264opts:0 su
bme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -f
orce_key_frames:0 \"expr:gte(t,0+n_forced*3)\" -sc_threshold:v:0 0 -vf \"setpara
ms=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw
\,ih*a)\,min(720\,304*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(720/a\,304))/2)*2,fo
rmat=yuv420p\" -codec:a:0 aac -ac 2 -ab 384000 -af \"volume=2\" -copyts -avoid_n
egative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_t
ime 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename \"/var/db/j
ellyfinserver/transcodes/89a47456a9b88c5d9f0dc472b6edf8ee%d.ts\" -hls_playlist_t
ype vod -hls_list_size 0 -y \"/var/db/jellyfinserver/transcodes/89a47456a9b88c5d
9f0dc472b6edf8ee.m3u8\""

Played and stopped gracefully. Some "Slow Http responses" though. I don't know what this means.

There's also ffmpeg transcode logs but I guess the relevant information is already above.

As I said before, I don't want to take more of your time, this was just meant as a status report; I'm happy with QSV and thank you for all the support you gave me.

I guess we should close the issue now, the main issue is solved.