pause/resume gstreamer stream when video stream stops/starts

[thiccaxe]

fixes some issues from #207

[fduncanh]

@thiccaxe

There are changes in pause/resume in gstreamer-1.24 (to do with locks) that breaks Uxplay (commenting out these pause/resume calls fixes the brokenness). See issue #284 I give details in the final comment.

(I found the place where the regression happen by finding which merge in gstreamer 1.24 development triggered the error in uxplay) There is probably a simple fix of your code in Uxplay to repair our use of pause/resume.

Arch is the only distro that has moved to gstreamer 1.24, which is a good place for testing.

• I also learned how to build , install (in the users home directory) and use specific merges (like "e506f9c2") of gstreamer from gitlab.gstreamer.org/gstreamer/gstreamer for testing purposes without destroying or removing the distribution-supplied gstreamer.

If you need details I can write up a note.

[thiccaxe]

Yep, the fix was a bit of a hack. I didn't really have an understanding of gstreamer when putting it together. What I guess might be happening is that earlier, when we paused it only paused the gstreamer internal clock. But the A/V in the buffer still flowed through. Then when it was unpaused there wouldn't be a delay. I think now the A/V buffer gets "locked?"

I will do some research. In the mean time keep an eye on Debian (rpi is based on this, not Ubuntu?). If it updates the version I suppose the commit will have to be reverted until a fix can be found. That, or better logic for when to play/pause.

[fduncanh]

Debian is more conservative. Debian Bookworm is 1.22.0. Debian Sid (Debian development) will be playing with gstreamer-1.24.

Arch is out "in front" with 1.24 immediate after its release. Last time I booted it, Manjaro (Arch derivative) was still in 1.22.10.

Ubuntu is Debian based, but takes stuff from Debian "Sid" as it matures. I suppose Debian Ubuntu 24.10 will have 1.24

The solution to the pause resume issue may be to delay sending the PPS + SPS NAL unit until the resume is confirmed. The symptom, of the issue are:

1. failure to find the Xwindow in 5 tries

2. failure to use the pps+sps nalu to start decoding the video stream

   failure 1 doesn't happen if uxplay is compiled without X11 support, so it is incidental.

(a different issue, invalid ntp time, may or may not be related to the pause/resume issue)

[fduncanh]

@thiccaxe

I added detection of GStreamer>=1.24 to the CMakelists.txt: GST_124 is defined if 1.24 or later is detected.

I thought that just enclosing the calls to video_pause, video_resume in #ifndef GST_124 ... #endif would be a temporary fix. but it seems not.

In any case, whatever the fix is, GST_124 works and can be used.

[fduncanh]

@thiccaxe

I released UxPlay-1.68.3 with a quick fix which just skips your code if gstreamer 1.24 or later is found at compile time.

I couldnt see any delays when I disabled the code (in the callbacks in uxplay.cpp) (I used an ipad not an iphone though) Maybe you could check if the original delay problem comes back if the code is disabled.

Instructions on how to trigger it would be useful if its still there

[thiccaxe]

Yep maybe those 3 commits from gst even fixed the problem altogether.

[fduncanh]

I didn't manage to see the issue using gstreammer-1.22.0 either. It would be good to check if its still there in that case.

Edit: I now saw it in 1.22.0

[thiccaxe]

Issue still seems to be there in 1.22.0 (1.22.9 specifically)

[thiccaxe]

how did you link the custom gstreamer version to uxplay? Some sort of flag on make or in the cmakelists?

Oh looks like it is as easy as using the "./gst-env.py" script

[fduncanh]

To build "my own" gstreamer-1.24, without killing my distro one:

I had to disable the gst-plugins-bad aja plugin

(1) uninstall gstreamer1.0-dev and gstreamer-plugins-base1.0-dev
(2)  update meson:  pip3 install meson     (install as user)
(3) git clone https://gitlab.freedesktop.org/gstreamer/gstreamer
(4) cd gstreamer; git checkout 1.24
(5) meson setup -Dprefix=/home/<username>/gstreamer-1.24 -Dgst-plugins-bad:aja=disable build
(6) cd build ; ninja
(7)   (after finished building) meson install
(8) edit /etc/ld.so.conf   to add /home/<username/gstreamer-1.24/lib     (and lib64)   ; sudo ldconfig
(9) export PATH=$HOME/gstreamer-1.24/bin:$PATH
(10)  rm -rf $HOME/.cache/gstreamer-1.0
(11) which gst-inspect-1.0       (should be the one you built)
(12) gst-inspect-1.0

now build UxPlay (pull latest)

export GSTREAMER_ROOT_DIR=$HOME/gstreamer-1.24/lib64       (path to gstreamer-1.0 directory)
cd 
cd UxPlay
cd build
cmake ..
make
ldd ./uxplay | grep gstreamer    should show its linked to your special gstreamer

[thiccaxe]

the fix is pretty simple we simply only pause if the current state is playing, and only resume if the current state is paused AND we have called the pause function before.

its not 100%, some small delay is there afterwards.

[fduncanh]

great!

[thiccaxe]

Nevermind, it all breaks if video orientation is changed.

[thiccaxe]

It's actually somewhat of a miracle that fix in #209 even worked (I never tested changing screen orientation, I don't know if you did). There is no way to actually know if the airplay client stops streaming except for checking for the absence of video packets.

[fduncanh]

as the client goes to sleep it seems to send an unencrypted PPS+SPS NAL unit, and send another one when it restarts

right now these NALs are held back and prepended to the next encrypted video data NAL. will need to check what happens in the code when two unencrypted NALs in a row are sent.

[fduncanh]

unfortunately Ubuntu-24.04 beta shows that it ships with gstreamer-1.24.2 and uxplay-1.68.2.

I posted a ubuntu bug report suggesting upgrade to 1.68.3

[thiccaxe]

Looks like Fedora 40 would still use 1.22. General release for 40 is still a while away as well.