Libreelec Raspberry openmax hw support

[brainstormi]

Hi... I'm trying to deploy jellyfin docker container to support gpu hw transcoding in Raspberry Pi 4B. Following jellyfin/linuxserver.io guidelines it's necessary to mount OpenMax libraries (/opt/vc/lib) inside the container to make it working and here it's where it's failing due to a linking error. When I execute the following command to create the container:

docker run -d --name jellyfin --volume /storage/docker/jellyfin:/config --volume /opt/vc/lib:/opt/vc/lib -p 8096:8096 --device=/dev/vchiq:/dev/vchiq --restart=unless-stopped linuxserver/jellyfin

Jellyfin container logs report the following error not being able to start the app web server:

[cont-init.d] 40-gid-video: executing... 
[jellyfin-init] Pi Libs detected loading
/sbin/ldconfig.real: Can't link /opt/vc/lib/libfdt.so.1 to libfdt.so
[cont-init.d] 40-gid-video: exited 0.
[cont-init.d] 99-custom-scripts: executing... 
[cont-init.d] 99-custom-scripts: exited 267.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

Library libftd.so is available in /opt/vc/lib in Libreelec filesystem: -rwxr-xr-x 1 root root 26328 Nov 23 00:46 /opt/vc/lib/libfdt.so

Does anybody know what it's failing mapping /opt/vc/lib volume?.

Regards,

[Artiume]

Hey, so I was the one who helped the lsio team get HWA support for the Rpi4. Our troubleshooting was based on raspbian. I'm on my phone so I can't get you the contents atm but look for a folder containing files such as libbcm_host.sc and libbrcmEGL.sc that'll be the proper folder.

[brainstormi]

Both libraries you pointed, in the case of Libreelec 9.2 distribution are located at /usr/lib folder... Indeed /opt/vc/lib is a symlink to /usr/lib. It doesn't matter which one of both I use for creating the container volume, the error is the same.

[thelamer]

Who built your distro libreelec ? your error indicates that the lib was compiled with the full soname /opt/vc/lib/libfdt.so.1 instead of a relative one.

[Artiume]

Looking at the lib issue, I don't have libfdt.so.1 (or anything similar) in my /opt/vc/lib on my raspbian OS

[brainstormi]

I burned the official image from Libreelec webpage. In /opt/vc/lib only appears libfdt.so, not libfdt.so.1

-rwxr-xr-x 1 root root 26328 Nov 23 00:46 /opt/vc/lib/libfdt.so

[Artiume]

Is thus your cross-post or just good timing lol https://forum.libreelec.tv/thread/21216-docker-jellyfin-hw-accelarated-transcoding-in-raspberry-pi/

[brainstormi]

Comparing with a Raspbian distribution, Libreelec /opt/vc/lib folder is pointing /usr/lib where OpenMax libraries are mixed with usual OS staff. Checking /etc/cont-init.d/40-gid-video file in this project, I see internally this volume mapping what does is preload all libraries available in /opt/vc/lib folder, but as this is pointing /usr/lib seems to make the things explode:

# openmax lib loading
if [ -e "/opt/vc/lib" ] && [ ! -e "/etc/ld.so.conf.d/00-vmcs.conf" ]; then
    echo "[jellyfin-init] Pi Libs detected loading"
    echo "/opt/vc/lib" > "/etc/ld.so.conf.d/00-vmcs.conf"
    ldconfig
fi  

[brainstormi]

Seems to be a priviledge issue. When executing ldconfig, container tries to create a symlink libfdt.so.1 -> libfdt.so in host /usr/lib (/opt/vc/lib -> /usr/lib) without writing privileges for that folder. Shouldn't the image have already installed OpenMax libraries instead of linking from host?

[thelamer]

We don't embed because it is not really a standard package, and a user should be using the sys level firmware from their exact install so we don't have versioning mismatches. /opt/vc/lib is the whole pi firmware stack or at least it is on almost all distros for the device not just openmax libs. I do not have a good answer for you, we do not want to load all the os libs from libreelec so using that dir is a dead end, we also do not want to maintain a list of actually needed libs to load as that can fluctuate in naming etc.

We definitely do not want to be changing host level symlinks for libs from our Docker container.

We have a team member that is on the LibreElec team, I pinged them but again I do not think their is a good solution to this.

[Artiume]

Could he pass the libs the same way that they're passed on raspbian? Overall, I hope we find a good solution since I'm looking forward to getting JF on libreelec as well :)

[brainstormi]

@thelamer I understand the reasons you exposed behind those decisions, thanks for your support.

In the meantime I tried to overcome this issue with Libreelec with a workaround. It seems that mapping manually the following two libraries do the trick...

 -v /opt/vc/lib/libopenmaxil.so:/opt/vc/lib/libopenmaxil.so 
 -v /opt/vc/lib/libbcm_host.so:/opt/vc/lib/libbcm_host.so 

It doesn't fully works, at least not though Jellyfin web UI. Now it seems it's hitting some kind of permissions issue with /dev/vchiq... If I try to transcode a file using web interface, ffmpeg crash just before starting transcoding with error 255 when running it under user 'abc':

[2020-01-21 17:29:23.446 +00:00] [ERR] FFMpeg exited with code 255 And in transcode log... it just stop just before starting transcoding:

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_omx))
  Stream #0:1 -> #0:1 (ac3 (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help

This doesn't happens if I manually launch the same ffmpeg command reported by the logs manually through container console under user root, doing the transcoding fine.

/usr/share/jellyfin-ffmpeg/ffmpeg -i file:"/media/XXX.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_omx -b:v 2552000 -maxrate 2552000 -bufsize 5104000 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "scale=trunc(min(max(iw\,ih*dar)\,1280)/2)*2:trunc(ow/dar/2)*2" -copyts -vsync -1 -codec:a:0 libmp3lame -ac 2 -ab 384000 -af "volume=2" -f hls -max_delay 5000000 -avoid_negative_ts disabled -start_at_zero -hls_time 3 -individual_header_trailer 0 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/data/transcoding-temp/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/a34e0a1b763209f083289c6f6a864ae5%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcoding-temp/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/a34e0a1b763209f083289c6f6a864ae5.m3u8"

But not sure about why, because checking the script /etc/cont-init.d/40-gid-video responsible of fixing the permission with vchiq mapped device seems to be working properly. If I check groups 'abc' user belongs to, he is already part of the 'root' group.

# groups abc
abc : abc root users

In Libreelec filesystem, doesn't exist 'video' group, so vchiq device belogs to root group:

crw------- 1 root root 245, 0 Apr 11 2019 /dev/vchiq

In the same way it's not a problem with the volumes mapped for transcoding, that already belongs to user abc and have 777 pattern as permission. What am I missing?

[Artiume]

If there's no video group and no normal user, that's an issue for LibreELEC. Does the OS run as root?

[brainstormi]

Yes, all processes included kodi.bin and dockerd are running with the root user.

[Artiume]

I'm half tempted to cut an issue ticket against them for the security risk, but I don't know how stripped down the OS is so I don't know how dangerous it is to run as root. That being said, I still don't like it >.>

[aptalca]

@brainstormi the issue you're facing seems to be that the file has no group read permissions set, so only root user can access it at the moment: crw------- 1 root root 245, 0 Apr 11 2019 /dev/vchiq

[Artiume]

Should be fixable by chown'ing it?

[aptalca]

Or chmod to add group read

[Artiume]

Yeah, it would be chmod, wouldn't it

[aptalca]

@Artiume libreelec is extremely stripped down. It's specifically made for kodi

@brainstormi if you're willing to test, I can put together a libreelec addon for jellyfin that sets it all up automatically. Let me know

[Artiume]

@aptalca Yeah, I've tried libreelec for a day since I am interested in its Rpi4 h265 decoding (I want to port this over for the RPi4 Jellyfin server). As a system, I don't like the idea of a linux box running native root, but I don't know how far down the system is stripped.

As for the /dev/vchiq properties, I'm seeing similar issues with DietPi. I had a user using it and we couldn't get it working for him.

@brainstormi if you're willing to test, I can put together a libreelec addon for jellyfin that sets it all up automatically. Let me know

This would be awesome :)

[brainstormi]

@Artiume libreelec is extremely stripped down. It's specifically made for kodi

@brainstormi if you're willing to test, I can put together a libreelec addon for jellyfin that sets it all up automatically. Let me know

No problem @aptalca , I can test it and provide feedback.

[aptalca]

@Artiume @brainstormi

Please download the test zip and install from zip: https://github.com/linuxserver/libreelec-addon-repo/blob/jellyfin/download/docker.linuxserver.jellyfin/docker.linuxserver.jellyfin-1.0.0.zip

Make sure you have first installed the linuxserver repo under le addons and add-on repository

In the addon settings, there is a toggle for openmax, enable it

You can watch the addon/container logs with journalctl -u docker.linuxserver.jellyfin -f

[Artiume]

Was checking out the docker command, putting everything on the docker bridge greatly reduces complexity, but would it be feasible to run it in host mode? Dlna and chromecast are known to show issues when using the docker bridge.

[aptalca]

Once we get hw transcode to work, I can add host networking as an addon setting.

For now I didn't because host would clash with emby (same ports)

[Artiume]

Fair enough :P no need to complicate things yet lol

[brainstormi]

It's failing from web and console too now, throwing the same error about not being able to find /opt/vc/lib/libbcm_host.so :

Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_omx)) Stream #0:1 -> #0:1 (ac3 (native) -> mp3 (libmp3lame)) Press [q] to stop, [?] for help [h264_omx @ 0x245bf80] /opt/vc/lib/libbcm_host.so not found Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height [libmp3lame @ 0x2462930] 3 frames left in the queue on closing Conversion failed!

[aptalca]

@brainstormi My bad, it was a silly typo. This should fix it: https://github.com/linuxserver/libreelec-addon-repo/raw/jellyfin/download/docker.linuxserver.jellyfin/docker.linuxserver.jellyfin-1.0.1.zip

[brainstormi]

No luck. Continue appearing the same error with the missing libbcm_host.so library. I tried to update the addon and also installing it from scratch, no way. I checked kodi was reporting correctly v1.0.1 for the addon. Inspecting the container filesystem, /opt is empty:

root@6d1daa245fa2:/opt# ls -la total 3 drwxr-xr-x 2 root root 1024 Feb 3 16:31 . drwxr-xr-x 1 root root 1024 Feb 14 16:40 ..

[aptalca]

After you install, can you check journalctl -u docker.linuxserver.jellyfin and see if it says /dev/vchiq not found, skipping openmax hw transcode?

[brainstormi]

It isn´t appearing in the logs...

# journalctl -u docker.linuxserver.jellyfin |grep vchiq
# 

[aptalca]

Hmm. Can you do docker inspect docker.linuxserver.jellyfin ?

And you did enable openmax in addon settings, right?

[brainstormi]

Output docker inspect docker.linuxserver.jellyfin: https://pastebin.com/ym1U3dF2 Openmax it's enabled, sure.

[aptalca]

I mean openmax enabled in the addon settings in libreelec, not jellyfin gui. Click on addons, my addons, services, and jellyfin and hit configure. You'll see the openmax toggle in there.

As long as the openmax toggle is enabled in libreelec addon settings, this code will check if vchiq exists. If so, it will map the necessary files and devices. If not, it will print a message about it in the addon log.

In your case, there are no files or devices mapped, and there is no message in the log, which leads me to believe openmax toggle is not enabled (unless there is another bug somewhere).

[brainstormi]

Excuse me, you're right. I wasn't aware about addon config itself and was only doing it only in Jellyfin through its web ui. Once enabled in the addon it's working perfect!!. Thank you very much for your support!

[aptalca]

Excellent. Thanks so much for testing. We'll go ahead and release it in the repo

[Artiume]

Awesome. Could you verify that jellyfin properly call for the openmax h264 and h265 decoders as well? I've been finding situations where HWA decoders not being called.

[brainstormi]

Seeing the logs, h264 seems to be correctly supported, at least it reports it's using h264_omx. In raspberry pi 4b consuming an average of 200% CPU, so there could be some paths of the processing that are done in software, like scaling, audio conversion... Regarding h265 I don't think it's supported in OMX and probably never will be: https://github.com/raspberrypi/firmware/issues/1168

[Artiume]

What library and api does libreelec use to decode h265? Libreelec uses ffmpeg to HWA decode it and that's what I'm interested in using to provide it in jellyfin on other OSes.

[brainstormi]

Some workaround with MMAL... The future looks like V4L2 would be the "standard"

[Artiume]

@aptalca you have any idea?