issue with patching video4linux2 for RPi 4B Ubuntu 22.04.1

(issue posted by @LockonS in #118 )

Not talking about the warnings. I follow the steps and run the debuild process. Plus, the gstreamer version on ubuntu 22.04 running on RPi 4B is 1.20.3. But I think that doesn't matter in this case.

apt list | grep gstreamer
# as can be seen the version is 1.20.3
gstreamer1.0-plugins-good/jammy-updates,now 1.20.3-0ubuntu1 arm64

Actually I want to talk about the problems I encountered and the incident that happend serveral times, which ultimately break the GUI environment in a mysterious way. Apology here as the topic might be a huge stray away from the point.

System image: ubuntu-22.04.1-preinstalled-desktop-arm64+raspi.img.xz (downloaded from ubuntu official site, using gdm3) Hardware: RaspberryPi 4B (4GB RAM)

As a conclusion, take a wild guess here that some package installed in order to get the patch done, breaks the Ubuntu GUI environment running on RPi 4B. Removing all the extra package doesn't fix the problem. I list all the packages installed with apt list --installed before and after the patch routine. After the GUI crashes, I start to uninstall all the extra packages with the two lists compared side by side. That doesn't fix the GUI.

Here is the detail about what happened and the steps I took.

Before trying to get the patch done, I build the gstreamer from source following the instruction. Strangely the gstreamer just installed doesn't support v4l2. As the output of gst-inspect-1.0 video4linux2, I didn't see the v4l2h264dec and v4l2h264enc. The complete output is listed here (Luckily I took an copy of the output at that time).

gst-inspect-1.0 video4linux2

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.20.3
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2022-06-15
  Binary package           GStreamer Good Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0

  v4l2deviceprovider: Video (video4linux2) Device Provider
  v4l2radio: Radio (video4linux2) Tuner
  v4l2sink: Video (video4linux2) Sink
  v4l2src: Video (video4linux2) Source

  4 features:
  +-- 3 elements
  +-- 1 device providers

Since this solution was not working, and I've got no idea why, I proceed to retry the patch solution. After a reboot, I start to run the patch routine and the steps I took was like below. Yes, at this time, the GUI environment is working as normal.

# installed a bunch of packages alone with devscripts and some of them might break the GUI
sudo apt-get install devscripts
sudo apt-get build-dep gstreamer1.0-plugins-good
# the first two steps went well, nothing strange happens
apt-get source gstreamer1.0-plugins-good
cd gst-plugins-good1.0-1.20.3
# now get to problem part, the debuild process fails
debuild

First time the debuild process failed with a list of unmet dependencies. After got them installed the process went on until another error occurs. A complete list of the dependencies installed in this step is listed here.

sudo apt-get install libavc1394-dev qtbase5-dev libjack-jackd2-dev libcaca-dev libpng-dev libshout3-dev libjpeg-dev libxext-dev libxfixes-dev libxv-dev qtbase5-private-dev qtdeclarative5-dev libqt5x11extras5-dev xvfb libxdamage-dev libjack-jackd2-dev debhelper libcairo2-dev libaa1-dev libgdk-pixbuf2.0-dev libgtk-3-dev libtag1-dev libpulse-dev libmpg123-dev libiec61883-dev

The second debuild error occurs when subprocess when ninja process was executing. Sorry I didn't backup the error output, and the searching history was gone with the system reinstall. I thought that debuild fail was no big deal and could be fixed following the error output. But I haven't tried since the GUI has crashed, I thought there is no point trying to get the patch done on that system environment, unless the GUI can be fixed. (Regrets. Should think that through and find the solution first, before I did another system clean install and wipe out everything. I felt like a fool 😭.)
As I tried to fix the problem occurs in the debuild process, then I found that GUI freezes with no response. No new windows can be open (Just like the previous time when I broke the GUI environment). Except for the already opened terminal windows. So I just reboot the system, and GUI failed to start as described.
After I compare the system journal and xession error log between an working GUI and the failed GUI, I noticed serveral lines of warning and error. And it seems to have something to do with the package libgdk-pixbuf2.0-dev, which got installed in step 3. The error info in $HOME/.xsession-errors also point to the package

# a part of system journal
Aug 27 13:11:56 dreamisty-node org.gnome.OnlineAccounts[1906]: /usr/libexec/goa-daemon: /usr/local/lib/aarch64-linux-gnu/libpng16.so.16: no version information available (required by /lib/aarch64-linux-gnu/libgdk_pixbuf-2.0.so.0)
Aug 27 13:11:56 dreamisty-node org.gnome.OnlineAccounts[1906]: /usr/libexec/goa-daemon: /usr/local/lib/aarch64-linux-gnu/libpng16.so.16: no version information available (required by /lib/aarch64-linux-gnu/libwebkit2gtk-4.0.so.37)
# the error in xsession-errors
/usr/libexec/gnome-session-binary: /usr/local/lib/aarch64-linux-gnu/libpng16.so.16: no version information available (required by /lib/aarch64-linux-gnu/libgdk_pixbuf-2.0.so.0)
/usr/libexec/gnome-session-binary: symbol lookup error: /lib/aarch64-linux-gnu/libgdk-3.so.0: undefined symbol: cairo_xlib_surface_get_display

Hopefully someone might be able to tell us what breaks the GUI environment and how to fix that, or at least prevent some others from wasting time on the same problem if they run Ubuntu on RPi 4B.

Originally posted by @LockonS in https://github.com/FDH2/UxPlay/issues/118#issuecomment-1229379821

@LockonS

I opened a new issue on this here. Will try to reproduce

downloaded 22.04.1 from ubuntu site https://ubuntu.com/download/raspberry-pi and installed with rapberry pi imager (note 22.04.1 LTS dated 2022-08-11 is also available directly via the imager, I would generally prefer to use that one, I don't know if there is any difference)
installed latest ubuntu-22.04.1 updates.
gst-inspect-1.0 video4linux2 shows

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.20.3
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2022-06-15
  Binary package           GStreamer Good Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0

  v4l2convert: V4L2 Video Converter
  v4l2deviceprovider: Video (video4linux2) Device Provider
  v4l2h264dec: V4L2 H264 Decoder
  v4l2h264enc: V4L2 H.264 Encoder
  v4l2jpegdec: V4L2 JPEG Decoder
  v4l2jpegenc: V4L2 JPEG Encoder
  v4l2radio: Radio (video4linux2) Tuner
  v4l2sink: Video (video4linux2) Sink
  v4l2src: Video (video4linux2) Source
  v4l2video18convert: V4L2 Video Converter

  10 features:
  +-- 9 elements
  +-- 1 device providers

this confirms that all the required kernel modules are present (unlike in the above report posted by @LockonS ). This R Pi 4B has had all needed firmware updates (it's at firmware version 000138a1, as shown by "sudo rpi-eeprom-update"). Maybe the one mentioned in issue #118 needs such an update? It might be missing the firmware needed by the bcm2835_codec.ko module? (To update the firmware, run "sudo rpi-eeprom-update -a", and reboot.)

lsmod | grep bcm reports

btbcm                  28672  1 hci_uart
bluetooth             716800  35 btrtl,btqca,btsdio,btintel,hci_uart,btbcm,bnep,rfcomm
bcm2835_v4l2           49152  0
bcm2835_codec          53248  0
bcm2835_isp            36864  0
snd_bcm2835            36864  1
videobuf2_vmalloc      20480  1 bcm2835_v4l2
v4l2_mem2mem           45056  1 bcm2835_codec
bcm2835_mmal_vchiq     40960  3 bcm2835_codec,bcm2835_v4l2,bcm2835_isp
videobuf2_dma_contig    24576  2 bcm2835_codec,bcm2835_isp
videobuf2_v4l2         32768  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,bcm2835_isp
videobuf2_common       81920  8 bcm2835_codec,videobuf2_vmalloc,videobuf2_dma_contig,videobuf2_v4l2,bcm2835_v4l2,v4l2_mem2mem,videobuf2_memops,bcm2835_isp
snd_pcm               163840  6 snd_usb_audio,snd_bcm2835,snd_soc_hdmi_codec,snd_compress,snd_soc_core,snd_pcm_dmaengine
videodev              282624  6 bcm2835_codec,videobuf2_v4l2,bcm2835_v4l2,videobuf2_common,v4l2_mem2mem,bcm2835_isp
mc                     73728  7 videodev,bcm2835_codec,snd_usb_audio,videobuf2_v4l2,videobuf2_common,v4l2_mem2mem,bcm2835_isp
vc_sm_cma              45056  2 bcm2835_mmal_vchiq,bcm2835_isp
snd                   126976  20 snd_seq,snd_seq_device,snd_hwdep,snd_usb_audio,snd_usbmidi_lib,snd_bcm2835,snd_soc_hdmi_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm,snd_rawmidi
bcm2835_gpiomem        16384  0
i2c_bcm2835            20480  0
spi_bcm2835            28672  0

confirming bcm2835_codec is available, so we can hope for a successful patching. (This is the kernel module that interfaces to the broadcom GPU firmware for hardware h264 video decoding, using video4linux2)

apt list --installed | grep gst shows that gstreamer1.0-plugins-good-1.20.3-0ubuntu1 from jammy-updates is installed. Some other gstreamer base packages are still at 1.20.1.
download THE CORRECT 1.20.3 PATCH (since jammy-updates uses plugins-good-1.20.3) wget https://github.com/FDH2/UxPlay/files/9196663/v4l2_1_20_3_diff3.txt.
The first step is to download the gstreamer1.0-plugins-good source package: edit /etc/apt/sources.list to uncomment the deb-src entries (or duplicate your deb entries and change "deb" to "deb-src") and then run sudo apt-get update. Then do

sudo apt-get install devscripts sudo apt-get build-dep gstreamer1.0-plugins-good

Next (in a directory where you will rebuild the package, and without using "sudo"), do

apt-get source gstreamer1.0-plugins-good cd gst-plugins-good1.0-1.20.3 debuild

This rebuilds the unpatched package. When all has finished, don't worry about a message that "running debsign failed".

OK: some of the tests run after rebuilding gstreamer1.0-plugins-good-1.20.3 seem to have failed. This is probably because Ubunta jammy-updates is mixing gstreamer-1.20.1 and 1.20.3 packages. (bad practice, should update everything to 1.20.3) The 3 failed tests were in elements flvmux, splitmuxsrc, jpegdec; also the camarabin test timed out. THESE ARE NOT THE VIDEO4LINUX2 elements we are trying to rebuild, so hopefully there is no problem. ("cd obj* ; ninja" returns ninja's message "nothing to do", which confirms that the recompilation was completed OK, it is just the tests of the rebuild of a few elements (that we don't care about) that failed, and that is unimportant).

now apply the patch and rebuild the patched video4linux2:do: (in this example, the patch was in the parent directory ..)

patch -p1 < ../v4l2_1_20_3_diff3.txt

The patch applied cleanly to 4 files: sys/v4l2/gstv4l2object.c, gstv4l2videodec.c and the corresponding .h include files

now rebuild and replace the installed libgstvideo4linux2.so:
```
cd obj*          
ninja
cd sys/v4l2
strip libgstvideo4linux2.so
sudo cp libgstvideo4linux2.so  /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
```
ninja succeeded, with the message "[24/24] Linking target sys/v4l2/libgstvideo4linux2.so", so we know it worked, then we strip libgstvideo4linux2.so, and overwrite it over the previously-installed version

"`ls -ltr /usr/lib/aarch64-linux-gnu/gstreamer-1.0/'" shows a new version of libgstvideo4linux2.so with size 321056.

reboot (probably not needed, but do it). Then install latest UxPlay 1.56 or github. (for the build, only needed to install cmake, libplist-dev, libavahi-compat-dnssd-dev ). For gstreamer plugins, install gstreamer1.0-plugins-bad, gstreamer1.0-libav.

UxPlay worked fine, no issues

@LockonS

I can't reproduce your issue, with a clean install of Ubuntu-22.04.1 64bit (jammy update) on R Pi Model 4B,

the "failure" of debuild of gstreamer-1.0-plugins-good-1.20.3 is only in testing some recompiled elements we don't care about. (I think the tests fail, because gstreamer1.0-plugins-base is left at 1.20.1 in Ubuntu jammy-update.) Just proceed to rebuild libgstvideo4linux2.so as descfribed. I'll update the writeup to say that the debuild testing failure is unimportant.
the significant issue in @LockonS 's system seems to be that "gst-inspect-1.0 video4linux2" seemed to show some firmware was missing/unupdated on his system.

Here is an article about updating the firmware https://all3dp.com/2/raspberry-pi-4-firmware-update-tutorial/

On my system, "sudo rpi-eeprom-update" reports my firmware on RPi Model 4B 8GB is up to date and is at version 000138a1 (although it also told me there was a newer bootloader available; this then updated when I ran "sudo rpi-eeprom-update -a" and then rebooted the system)

OMG! Upgrading the firmware worked like a charm! I did the firmware upgrade and now the firmware is at version 000138a1. But, it seems bootloader is what had been updated. Now that uxplay work fine and no GUI environment broken! Thank you so much! I suspect that some weird bugs happened after some package installed in Ubuntu 22.04 running with an older bootloader. Better remind the others to upgrade the firmware first?

```shell
# before upgrade
sudo rpi-eeprom-update

*** UPDATE AVAILABLE ***
BOOTLOADER: update available
   CURRENT: Tue Jul  6 10:44:53 AM UTC 2021 (1625568293)
    LATEST: Tue Jan 25 02:30:41 PM UTC 2022 (1643121041)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Dedicated VL805 EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1

# after upgrade
sudo rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Tue Jan 25 02:30:41 PM UTC 2022 (1643121041)
    LATEST: Tue Jan 25 02:30:41 PM UTC 2022 (1643121041)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Dedicated VL805 EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1
```

Yet I noticed another problem with the option -v4l2. It seems with that option, hardware accerlation is enabled but I've got a worse video quality? I'll do some comparsion with the previous worked 3B+ and post a new issue.

yes -v4l2 may actually degrade latency, it seems to depend on which linux distribution is being run (maybe on how they compiled the kernel) sometime just no options is best , and autovideosink finds the best pipeline. Other times you do better with -v4l2.

Also experiment with -vs glimagesink

The patching instructions in the Wiki have been updated based on your experience

Thanks for the effort~

FDH2 / UxPlay

issue with patching video4linux2 for RPi 4B Ubuntu 22.04.1 #123