Make project deployable with balena

[iayanpahwa]

It would be really amazing to have this project available to deploy on Single board computers like Raspberry Pi using balena. This will help users to directly use deploy-with-balena button from README and have it running with ease. They can even use balenaCloud dashboard to manage the fleet of devices using this project.

There is a similar project on balenaHub called rpi-play, which is an open-fleet being used by ~800 people at the moment, since UxPlay is more actively maintained it would be great to replace it with this one or both can co-exist.

I am happy to help with the contributions required to balenafy UxPlay 🙂

[fduncanh]

what needs to be done?

Uxplay is derived from RPiPlay, with various improvements (extra features, fixes) but uses GStreamer for audio and video rendering (with Video4Linux2 for the Raspberry Pi GPU video h264 decoding for video and audio rendering, rather than the deprecated OpenMAX used by RPiPlay). It was oriented toward desktop Linux, but now also supports Raspberry Pi.

One issue for RPI users is that right now the GStreamer Video4linux2 plugin needs to be patched with backports from GStreamer-1.21 development, which will be in the 1.22 release scheduled for this summer. (Raspberry Pi OS has released a partial backport to 1.18.4 and I was told the the rest of the backport I suggested was in the queue waiting for release.) GStreamer 1.20.2 may also have some of the backport when it is released.

A further issue is the the update by Rasperry Pi OS from kernel 5.10 to 5.15 has broken something (probably in OpenGL) used in the "Desktop" version with X11, but the "Lite" version with kms graphics is still working just fine with UxPlay and 5.15 kernel.

These seem to be temporary problems in the abandonment of 32-bit OpenMAX (which could obsolete RPiPlay) and its replacement by Video4Linux2 for the Broadcom GPU hardware video decoding needed for the R Pi to be a viable 64-bit audiovisual platform.

[rahul-thakoor]

hello @fduncanh

I published https://hub.balena.io/gh_rahul_thakoor/balena-rpiplay

I can help here if needed. Thanks for providing such a detailed description current implementation and constraints.

Some points to note:

• as of today, balenaOS releases are still on 5.10 kernel
• We can have specific dockerfiles for raspberry pi and then include the backports as required
• we would prefer the "lite: version anyway.

I can help with creating the Dockerfiles and "Deploy with balena" buttons.

We can then update the balena-rpiplay fleet on balenaHub to use this project instead.

let me know your thoughts on this. Thanks

[fduncanh]

UxPlay is GPLv3, so you are free to do what you want with it (subject to GPL). Feel free to experiment with a balena package! You are the balena expert! There don't seem to be any balena-specific changes on the RPiPlay site, so it looks like you don't need anything to be done here? feel free to ask for any advice.

The links to the GStreamer patching issue are in the UxPlay README: that will be the main balena difficulty, I would think. https://github.com/FDH2/UxPlay/wiki/Gstreamer-Video4Linux2-plugin-patches

[rahul-thakoor]

hey @fduncanh

what are the runtime dependencies for uxplay on raspberry pi?

should gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good be ok?

[fduncanh]

For the Lite, yes. For the "desktop" (with X11) add gstreamer1.0-gl Also libssl1.1 (for libcrypto.so.1.1) and libplist3 (for libplist-2.0.so.3) and avahi (libavahi-common.so.3 , libavahi-client.so.3 libdns_sd.so.1)

from the list below (ldd) which doesnt include gstreamer plugins: libavahi-compat-libdnssd1 libplist3 libssl1.1 libgstreamer-plugins-base1.0 libgstreamer1.0 libglib2.0-0 libavahi-common3 libavahi-client3

On a 32 bit Lite system:

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.15.30-v7l+ #1536 SMP Mon Mar 28 13:51:42 BST 2022 armv7l GNU/Linux
pi@raspberrypi:~ $ ldd /usr/local/bin/uxplay
        linux-vdso.so.1 (0xbef87000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0xb6f7b000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6f40000)
        libplist-2.0.so.3 => /lib/arm-linux-gnueabihf/libplist-2.0.so.3 (0xb6f21000)
        libcrypto.so.1.1 => /lib/arm-linux-gnueabihf/libcrypto.so.1.1 (0xb6d0a000)
        libdns_sd.so.1 => /lib/arm-linux-gnueabihf/libdns_sd.so.1 (0xb6cf2000)
        libgstapp-1.0.so.0 => /lib/arm-linux-gnueabihf/libgstapp-1.0.so.0 (0xb6cd3000)
        libgstreamer-1.0.so.0 => /lib/arm-linux-gnueabihf/libgstreamer-1.0.so.0 (0xb6b99000)
        libgobject-2.0.so.0 => /lib/arm-linux-gnueabihf/libgobject-2.0.so.0 (0xb6b3d000)
        libglib-2.0.so.0 => /lib/arm-linux-gnueabihf/libglib-2.0.so.0 (0xb6a1a000)
        libstdc++.so.6 => /lib/arm-linux-gnueabihf/libstdc++.so.6 (0xb6892000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6823000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb67f6000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb66a2000)
        /lib/ld-linux-armhf.so.3 (0xb6f90000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb668e000)
        libavahi-common.so.3 => /lib/arm-linux-gnueabihf/libavahi-common.so.3 (0xb6673000)
        libavahi-client.so.3 => /lib/arm-linux-gnueabihf/libavahi-client.so.3 (0xb6654000)
        libgstbase-1.0.so.0 => /lib/arm-linux-gnueabihf/libgstbase-1.0.so.0 (0xb65cd000)
        libgmodule-2.0.so.0 => /lib/arm-linux-gnueabihf/libgmodule-2.0.so.0 (0xb65b9000)
        libunwind.so.8 => /lib/arm-linux-gnueabihf/libunwind.so.8 (0xb6583000)
        libdw.so.1 => /lib/arm-linux-gnueabihf/libdw.so.1 (0xb64e3000)
        libffi.so.7 => /lib/arm-linux-gnueabihf/libffi.so.7 (0xb64cb000)
        libpcre.so.3 => /lib/arm-linux-gnueabihf/libpcre.so.3 (0xb6454000)
        libdbus-1.so.3 => /lib/arm-linux-gnueabihf/libdbus-1.so.3 (0xb6400000)
        liblzma.so.5 => /lib/arm-linux-gnueabihf/liblzma.so.5 (0xb63cf000)
        libelf.so.1 => /lib/arm-linux-gnueabihf/libelf.so.1 (0xb63a6000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb637e000)
        libbz2.so.1.0 => /lib/arm-linux-gnueabihf/libbz2.so.1.0 (0xb635e000)
        libsystemd.so.0 => /lib/arm-linux-gnueabihf/libsystemd.so.0 (0xb62a7000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xb628f000)
        libzstd.so.1 => /lib/arm-linux-gnueabihf/libzstd.so.1 (0xb61c2000)
        liblz4.so.1 => /lib/arm-linux-gnueabihf/liblz4.so.1 (0xb6195000)
        libgcrypt.so.20 => /lib/arm-linux-gnueabihf/libgcrypt.so.20 (0xb60b8000)
        libgpg-error.so.0 => /lib/arm-linux-gnueabihf/libgpg-error.so.0 (0xb608b000)

[rahul-thakoor]

Hey

Yes, I am building the "lite" version and intend to use kmssink

from the list below (ldd) which doesnt include gstreamer plugins: libavahi-compat-libdnssd1 libplist3 libssl1.1 libgstreamer-plugins-base1.0 libgstreamer1.0 libglib2.0-0 libavahi-common3 libavahi-client3

Installing these and the gstreamer plugins, the current image size is around 1GB. I would like to use tools like https://github.com/larsks/dockerize or https://github.com/intoli/exodus to create a leaner image and only include required artifacts. Is there a way I can patch CMakeLists.txt such that ldd picks up the required gstreamer plugins as dependencies? checking here it seems they are required anyway.

Thanks

[fduncanh]

Looking at the code, as compared to rpiplay, I see (I followed and modified the antimof code) I dont have a video plugin check, just an audio one, while rpiplay seems to have added one for video too.

The plugin check is not in fact necessary,its to alert the user about missing plugins. and I dont think it is even complete (and may be overcomplete).

The plugins are NOT dependencies of uxplay. There are used by gstreamer to assemble the video and audio pipelines.

uxplay does the pairing with the Apple device, and decrypts the audo and video streams, and pushes them into the gstreamer pipelines as bitstreams. gstreamer takes over after that.

Is there any prior history of gstreamer use with balena?

In principle, there maybe be should be two balena packages, uxplay and gstreamer, with gstreamer providing the audio and video rendering services for the bitstreams produced by uxplay.

[fduncanh]

I looked at the history of the plugin checks in rpiplay.

The check from the audio renderer got duplicated in the video renderer part when the backport from uxplay was done, and an extra check for the h264 parser was added. I'll think about this issue, (and maybe add a check for the parser to uxplay),

In the meantime just get things working without worrying about creating a leaner image, and tackle that after it is working.

[rahul-thakoor]

The plugins are NOT dependencies of uxplay. There are used by gstreamer to assemble the video and audio pipelines.

thanks for the clarification :)

Is there any prior history of gstreamer use with balena?

afaik, several projects use gstreamer on balena. here is one example.

In principle, there maybe be should be two balena packages, uxplay and gstreamer, with gstreamer providing the audio and video rendering services for the bitstreams produced by uxplay.

interesting! this sounds like we can have a generic gstreamer service which we can publish as a block on hub. I'm not too familar on how to applications can interface with gstreamer. I will read up a bit 👍

[rahul-thakoor]

Running uxplay, server starts but when I connect, it disconnects immediately:

19.04.22 13:45:37 (+0000) Initialized server socket(s)
19.04.22 14:15:07 (+0000) Accepted IPv4 client on socket 26
19.04.22 14:15:07 (+0000) Local: 192.168.100.14
19.04.22 14:15:07 (+0000) Remote: 192.168.100.8
19.04.22 14:15:07 (+0000) Open connections: 1
19.04.22 14:15:07 (+0000) Client identified as User-Agent: AirPlay/610.19.1
19.04.22 14:15:07 (+0000) Accepted IPv4 client on socket 28
19.04.22 14:15:07 (+0000) Local: 192.168.100.14
19.04.22 14:15:07 (+0000) Remote: 192.168.100.8
19.04.22 14:15:07 (+0000) Open connections: 2
19.04.22 14:15:07 (+0000) raop_rtp_mirror starting mirroring
19.04.22 14:15:08 (+0000) Begin streaming to GStreamer video pipeline

uxplay startup was  good.,
GStreamer now fails to assemble a working pipeline.

19.04.22 14:15:08 (+0000) GStreamer error: Internal data stream error.
19.04.22 14:15:08 (+0000) Removing connection for socket 26
19.04.22 14:15:08 (+0000) Destroying connection
19.04.22 14:15:08 (+0000) Open connections: 1
19.04.22 14:15:08 (+0000) raop_rtp_mirror->running is no longer true
19.04.22 14:15:08 (+0000) Removing connection for socket 28
19.04.22 14:15:08 (+0000) Destroying connection
19.04.22 14:15:08 (+0000) Open connections: 0
19.04.22 14:15:08 (+0000) Initialized server socket(s)
19.04.22 14:15:19 (+0000) Accepted IPv4 client on socket 26
19.04.22 14:15:19 (+0000) Local: 192.168.100.14
19.04.22 14:15:19 (+0000) Remote: 192.168.100.8
19.04.22 14:15:19 (+0000) Open connections: 1
19.04.22 14:15:19 (+0000) Client identified as User-Agent: AirPlay/610.19.1
19.04.22 14:15:19 (+0000) Accepted IPv4 client on socket 28
19.04.22 14:15:19 (+0000) Local: 192.168.100.14
19.04.22 14:15:19 (+0000) Remote: 192.168.100.8
19.04.22 14:15:19 (+0000) Open connections: 2
19.04.22 14:15:20 (+0000) raop_rtp_mirror starting mirroring
19.04.22 14:15:20 (+0000) Begin streaming to GStreamer video pipeline

using uxplay -rpi -vs kmssink -d I get:

supported audio format 1: AAC-ELD 44100/2
supported audio format 2: ALAC 44100/16/2
supported audio format 3: AAC-LC 44100/2
supported audio format 4: PCM 44100/16/2 S16LE
GStreamer video pipeline will be:
"appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true ! queue ! h264parse ! capssetter caps="video/x-h264, colorimetry=bt709" ! v4l2h264dec ! v4l2convert ! kmssink name=video_sink sync=false"
uxplay: /usr/src/app/uxplay-source/renderers/video_renderer_gstreamer.c:149: video_renderer_init: Assertion `renderer->sink' failed.
Aborted (core dumped)

any help is much appreciated :pray:

[fduncanh]

uxplay itself worked fine, but Gstreamer was unable to find a working pipeline.

get some GStreamer debug info with

export GST_DEBUG=2

before running uxplay.

GST_DEBUG=4 will give a LOT more info.

with uxplay-1.50 you just need uxplay -rpi (without the -vs kmssink).

[fduncanh]

also get the output of gst-inspect-1.0 video4linux2 gst-inspect-1.0 h264parse

[fduncanh]

@rahul-thakoor any progress?

If you have something installable on a Pi but not working yet, I can take a look.

[rahul-thakoor]

Hi

Yes, I can successfully mirror from my iPhone.

I can't get audio to work yet. I am trying with pulseaudio backend. Will get back on this soon

Thanks

[fduncanh]

yes pulse audio is needed on the PI, at least for digital sound. (-as pulsesink is the uxplay option if you dont leave it to autoaudiosink )

[fduncanh]

this seems to be adandoned? shall I close the issue?

[rahul-thakoor]

hey sorry I haven't got a chance to continue working on this. you can close for now, i will add details once I get back to it. thanks

[fduncanh]

sure, feel free to ask for any information you need whenever!