search

FDH2 / UxPlay

AirPlay Unix mirroring server
GNU General Public License v3.0
1.35k stars 72 forks source link

Latency #66

Closed xXPerditorXx closed 2 years ago

xXPerditorXx commented 2 years ago

At first the Latency between my ipad and UxPlay is like 1 second but then goes up to more than 2 minutes.

Tested on raspberry pi 4 8gb and Arch

fduncanh commented 2 years ago

Definitely not correct. 1 sec or 0.5 sec could be expected. But uxplay is not so far tested on RPi 4 (a 4B 8GB is on its way, just to test)

UxPlay is aimed at x86 architectures, but (according to RPi4 specs) should work OK on Pi 4.

The key is getting GStreamer to use the built-in hardware h264 decoding on Pi 4. (I don't yet know how this is done, you need advice from Pi4 users) see #65

do the following tests.

  1. look at the output of

    uxplay  -d

  2. look at the output of

    export GST_DEBUG=2 
uxplay
fduncanh commented 2 years ago

what is the operating system? (and version) 32 or 64 bit (both OK for PI 4 8GB, 64bit is not necessarily better, since some things are not ported to 64bit yet) EDIT OK i see its arch.

xXPerditorXx commented 2 years ago

Raspberry Pi OS 64 bit and 32bit

uxplay -d show many of these:

raop_rtp_mirror video ntp = 1645130213432792, now = 1645130213362416, latency = -70376
raop_rtp_mirror video ntp = 1645130213449468, now = 1645130213363276, latency = -86192
raop_rtp_mirror video ntp = 1645130213466144, now = 1645130213375605, latency = -90539
raop_rtp_mirror video ntp = 1645130213482820, now = 1645130213394469, latency = -88351

And short after that:

raop_rtp_mirror video ntp = 1645130214400009, now = 1645130214341255, latency = -58754
raop_rtp_mirror video ntp = 1645130214416685, now = 1645130214496958, latency = 80273
raop_rtp_mirror video ntp = 1645130214433362, now = 1645130214498020, latency = 64658
raop_rtp_mirror video ntp = 1645130214450038, now = 1645130214498844, latency = 48806
raop_rtp_mirror video ntp = 1645130214466714, now = 1645130214499611, latency = 32897
raop_rtp_mirror video ntp = 1645130214483390, now = 1645130214500444, latency = 17054
raop_rtp_mirror video ntp = 1645130214500066, now = 1645130214553808, latency = 53742
raop_rtp_mirror video ntp = 1645130214516742, now = 1645130214555968, latency = 39226
raop_rtp_mirror video ntp = 1645130214533419, now = 1645130214583184, latency = 49765
raop_rtp_mirror video ntp = 1645130214550095, now = 1645130214585449, latency = 35354
raop_rtp_mirror video ntp = 1645130214566771, now = 1645130214587630, latency = 20859
raop_rtp_mirror video ntp = 1645130214583447, now = 1645130214589763, latency = 6316
raop_rtp_mirror video ntp = 1645130214616799, now = 1645130214592889, latency = -23910
fduncanh commented 2 years ago

These all look good (working correctly) : latency is in microseconds, 80531 = 0.08 sec 23910 = 0.024 sec (1/40 sec)

two minutes latency would be 120000000

EDIT this is not necessarily the latency you see, it's measured BEFORE the video and audio is rendered by GStreamer. If you are using software h264 decoding because the hardware h264 decoder in the RPi GPU is not being accessed , the observed latency could be much bigger, as the R Pi 4 CPU may not be powerful enough for fast software decoding..

fduncanh commented 2 years ago

Did you manage to get the Pi to use its v4l2 hardware h264 decoding? (on the broadcom chip)

Any advice about this for #65 would be appreciated ( of course you use arch linux and #65 is using Raspberry OS)

fduncanh commented 2 years ago

You seem to be working on Raspberry Pi 8GB, at least partially. (when does latency become bad?)

The interesting debug is not the uxplay -d but both levels of GStreamer debug output: (please post here!)

export GST_DEBUG=2
uxplay

and

export GST_DEBUG=4
uxplay

also look at

uxplay -avdec

(force software h264 decoding; maybe also reduce fps with uxplay -fps 15 -avdec)

and (on latest github)

uxplay -v4l2

(undocumented option, tries to force RPi hardware h264 decoding, don't know if it works, as dont yet have PI 4)

also, this should tell you if your GStreamer setup can see the PI's v4l2 hardware decoder. (some steps to activate this are needed)

gst-inspect-1.0 video4linux2

see https://stackoverflow.com/questions/64426904/no-such-element-or-plugin-v4l2h264enc

xXPerditorXx commented 2 years ago

The latency comes when I start an app or game. The latency I mean is something like slow motion. When I do one of the things described above, the latency gets high and something like a slow motion occurs. Even after the app is closed, the latency remains as high, only this slow motion disappears a bit.

Following command is used: uxplay -p tcp 36836 -p udp 36830 -n '404 NotFound' -nh -fps 60

GST_DEBUG=2:

using network ports UDP 36830 36831 36832 TCP 36836 36837 36838
using system MAC address dc:a6:32:b6:10:e2
Initialized server socket(s)
Accepted IPv4 client on socket 27
Local: 192.168.168.253
Remote: 192.168.168.34
Open connections: 1
Client identified as User-Agent: AirPlay/610.18.1
Accepted IPv4 client on socket 29
Local: 192.168.168.253
Remote: 192.168.168.34
Open connections: 2
raop_rtp_mirror starting mirroring
Begin streaming to GStreamer video pipeline
0:00:13.618999859 64439   0x556f3c9e40 WARN               decodebin gstdecodebin2.c:2389:connect_pad:<decodebin0> Element v4l2h264dec0 does not accept caps
ct=8 spf=480 usingScreen=1 isMedia=1  audioFormat=0x1000000
start audio connection, format AAC-ELD 44100/2
raop_rtp starting audio
0:00:24.289571827 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:24.889141390 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:25.969093144 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:27.051076284 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:28.109095233 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:30.159136606 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:33.329122575 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:33.688859426 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:35.309011462 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:36.229597509 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:36.888948946 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
0:00:44.318956423 64439   0x7f84008100 WARN                   pulse pulsesink.c:702:gst_pulsering_stream_underflow_cb:<autoaudiosink0-actual-sink-pulse> Got underflow
raop_rtp_mirror: prepended sps_pps timestamp does not match that of video payload
raop_rtp_mirror tcp socket closed
Connection closed for socket 27
Destroying connection
Open connections: 1
Connection closed for socket 29
Destroying connection
Open connections: 0

GST_DEBUG=4: uxplay-GST_DG4.txt I marked with '---' where I started the game, closed the game, disconnected and pressed Ctrl + C

With -avdec (uxplay -p tcp 36836 -p udp 36830 -n '404 NotFound' -nh -fps 15 -avdec) the home screen was impressive fast, almost realtime but the frames have to be fluently. HTOP shows the hole time the following (60fps with avdec): image

uxplay -v4l2 is not working:

pi@raspberrypi-8GB:~ $ uxplay -p tcp 36836 -p udp 36830 -n '404 NotFound' -nh -fps 60 -v4l2
using network ports UDP 36830 36831 36832 TCP 36836 36837 36838
using system MAC address dc:a6:32:b6:10:e2
Initialized server socket(s)
Accepted IPv4 client on socket 29
Local: 192.168.168.253
Remote: 192.168.168.34
Open connections: 1
Client identified as User-Agent: AirPlay/610.18.1
Accepted IPv4 client on socket 31
Local: 192.168.168.253
Remote: 192.168.168.34
Open connections: 2
raop_rtp_mirror starting mirroring
Begin streaming to GStreamer video pipeline
GStreamer error: Internal data stream error.
Removing connection for socket 29
Destroying connection
Open connections: 1
Removing connection for socket 31
Destroying connection
Open connections: 0
Initialized server socket(s)

gst-inspect-1.0 video4linux2:

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.18.4
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2021-03-15
  Binary package           GStreamer Good Plugins (Debian)
  Origin URL               http://packages.qa.debian.org/gst-plugins-good1.0

  v4l2src: Video (video4linux2) Source
  v4l2sink: Video (video4linux2) Sink
  v4l2radio: Radio (video4linux2) Tuner
  v4l2deviceprovider: Video (video4linux2) Device Provider
  v4l2jpegdec: V4L2 JPEG Decoder
  v4l2h264dec: V4L2 H264 Decoder
  v4l2h264enc: V4L2 H.264 Encoder
  v4l2convert: V4L2 Video Converter
  v4l2video18convert: V4L2 Video Converter

  9 features:
  +-- 8 elements
  +-- 1 device providers
fduncanh commented 2 years ago

Good: the v4l2h264dec H264 hardware decoder is shown as available. Have you tried the (hidden) -v4l2 option for this I added a few days ago? The audio should be pulseaudio, I believe 

uxplay -v4l2 -vs glimagesink -as pulsesink

might be optimal on the PI -fps wont increase the framerate, it can just throttle it to below 30fps I am soon getting a R PI 4B 8GB and will experiment.

EDIT: -avdec uses the CPU for h264 decoding, but when the CPU was shared with other tasks you got latency. -v4l2 (hopefully) uses the GPU (not yet tested by me)

fduncanh commented 2 years ago

The latest github (which will eventually be uxplay 1.48) allows complete control of the video pipeline with three options:

uxplay  -vd  <video h264 decoder> -vc <video_converter> -vs <videosink>

the pipeline is .... ! h264parse ! <video h264 decoder> ! <video converter> ! <videosink>

uxplay -d will show the pipeline that was created with these options. quotes " ... " can be used for including parameters

I now have a R Pi 4B 8GB, (raspi OS bullseye) but I haven't worked out how to get hardware decoding working. Using avdec_h264 software decoding, the latency is pretty bad.

Please report any successes.

fduncanh commented 2 years ago

opening a new issue for the Raspberry PI

fduncanh commented 2 years ago

@xXPerditorXx Uxplay now supports R Pi with hardware GPU video decoding. (you need to patch GStreamer, until updates are available from distributions)

see the latest README

xXPerditorXx commented 2 years ago

How do you fixed it?

fduncanh commented 2 years ago

it was difficult.

gstreamer v4l2 for using the broadcom GPU in the pi was broken in two ways one way I found a(or adapted someone else's )fix, the other we got some action from the gstreamer devels about. turns out a fix was around but not committed in the development branch. Its only the decision of R Pi to drop omx that has made the issue of fixing v4l2 urgent.

Now it is in the development branch, and backports to 1.18.4 and 1.20.0 are available from this site (see issues) as a patch until it comes out in a gstreamer release. The R Pi OS will probably have the patch soon, but you can get it here now)