(Windows issues, some now fixed): Is it possible to set a buffer/delay?

WinkelCode commented 1 year ago

I have two problems:

When only streaming audio, it's a bit too fast when compared to the Apple Music lyrics, my iPad assumes a delay that isn't there.
The video is a bit choppy when connected via WLAN, I tried with a LAN adapter on my iPad and it seemed smoother. I suspect that a larger buffer may also help here.

I can't test video with it, but Shairport4w has an option to set the buffer size, a size of 201 frames seems perfect, at least for the music delay.

Is there such an option for UxPlay? I couldn't find anything.

fduncanh commented 1 year ago

-ao seems good.

fduncanh commented 1 year ago

@WinkelCode @thiccaxe

option -ad n has been removed and replaced with option -ao x where x is a decimal (like 5 or 4.98, in the range [0.0, 10.0] in seconds (will be converted to a whole number of milliseconds).

Its now documented on manpage , README, etc., and marks the beginning of uxplay 1.62 development.

thiccaxe commented 1 year ago

Might want to consider moving to discussions

fduncanh commented 1 year ago

need more guidance on these Bug reports from @WinkelCode (have not reproduced them so far)

If the default audio format isn't set to 44100Hz, audio playback has popping like an old record.
Video performance could be better, weird stutters/bad frame timing (approx. first <30 secs or so?), compression artifacts (might be normal?) I wonder if bad frame timing issues could be related to, as previously discovered, the NTP data being apparently ignored? Maybe the latency variation gets naturally better the longer it is connected?
Weird audio artifacting when changing volume in screen mirror mode.

fduncanh commented 1 year ago

@WinkelCode

I think I have identfied the issue with artefacts when changing the volume

in lib/raop_rtp.c line 345:

 /* Call set_volume callback if changed */
    if (volume_changed) {
        raop_buffer_flush(raop_rtp->buffer, flush);     <=============== line 345
        if (raop_rtp->callbacks.audio_set_volume) {
            raop_rtp->callbacks.audio_set_volume(raop_rtp->callbacks.cls, volume);
        }
    }

The history shows that the RPiPlay developer FD- added this raop_buffer_flush on July 12 2019 https://github.com/FD-/RPiPlay/commit/ad0872961fc34c3b86eae1ba08e6b820f5ff10bc

There seems absolutely NO reason to flush the raop audio buffer when the volume is changed (Volume changes happen in gstreamer), and this raop buffer is for audio decryption before sending to gstreamer. So I think this is just a mistake.

Can you ~comment out this line raop_buffer_flush(raop_rtp->buffer, flush); and recompile to~ see if it fixes the problem you hear (without screwing up anything else) Thanks!

EDIT I tested and committed this change now. I think it fixes the volume-change issues. Please verify!

WinkelCode commented 1 year ago

need more guidance on these Bug reports from @WinkelCode (have not reproduced them so far)

If the default audio format isn't set to 44100Hz, audio playback has popping like an old record.

To my surprise, it is actually possible to record this problem (upload at 48kHz, I can still hear it). Good headphones and hearing may be required.

resampling bug.webm

Video performance could be better, weird stutters/bad frame timing (approx. first <30 secs or so?), compression artifacts (might be normal?) I wonder if bad frame timing issues could be related to, as previously discovered, the NTP data being apparently ignored? Maybe the latency variation gets naturally better the longer it is connected?ing when changing volume in screen mirror mode.

I did some more testing just now:

The macOS Airplay server also has the initial stuttering
The macOS Airplay server has a noticeably larger buffer (longer latency until shown) than UxPlay
Setting -fps 60 eliminates most frame-timing problems and reduces compression artifacting, and should probably be the default.
Stuttering beyond the first few seconds may be dependent on client system load: when the stream stutters, the iPad is still smooth, or stutters much less so.
Network congestion may also cause stuttering or frame-timing issues, macOS will have a large buffer for a reason.
I will have to do more comparisons between my MacBook and UxPlay in that regard. with -fps 60, most issues seem to be resolved, but I am not sure if the rest is all from the client, or if some improvement can be gained with buffering like macOS does.

Here is a video of the initial stuttering. Notes: For our purposes, the "stutter warning" from the TestUFO website can be ignored, a few stutters can also be seen when it already says "Ready*".

https://user-images.githubusercontent.com/29005772/212653064-df17f37e-5851-42e8-8525-3c285b250b76.mp4

EDIT I tested and committed this change now. I think it fixes the volume-change issues. Please verify!

Fixed for me as well.

fduncanh commented 1 year ago

Lots of progress with your buglist! thanks for spotting them!

I increased the maxFPS default (controlled by -fps option) to 60

The fps work is all mine, I cannot blame previous authors. The original plist entries in AirServer and RPiPlay were given as a binary dump from an apple TV 3,1, which showed maxFPS=30. The RPiPlay author later converted these settings to plist form, and accidentally omitted the maxFPS setting (or confused it with refresh rate, and gave it incorrectly in AirPlay1 format): it tells the client the maximum fps it is allowed to send at. I added the -fps option to allow the user to change maxFPS

Only later, by examining all transmissions for the client, did I discover that the client sends fps data once per second to server, and exposed this with the -FPSdata option (these were mystery transmissions that earlier code ignored)

Early testing with gstreamer's -vs fpsdisplaysink showed that ~average fps did not exceed 30 ~ even with -fps 60, but now with -FPSdata I see bursts of higher fps which would be impeded by -fps 30. EDIT Now I see 4K youtube videos do play at 60fps on UxPlay with -fps 60 (my error in keeping this setting from an ancient appleTV setting!)

This shows at least some transmission above 30fps when -fps60 is used. So I have now reset the default maxFPS to 60. may this is the best fix that can be obtained?

EDIT use -vs fpsdisplaysink to monitor current and average fps

fduncanh commented 1 year ago

@WinkelCode

I did some more cleanups on the PTS ( gstreamer presentation timestamps) which I think are being used but I still am not entirely sure. Before they were really broken because I hadn't converted from usec to nsec. But then, how was video and audio getting synced? ( Mysteries of GStreamer....)

Now the huge unix-time offset is gone, and "monotonic time" is being used, with the PTS measured in nsecs from the starting times of their respective (audio and video) pipelines.

This "issue" here is too long now, and your comments inspired a lot of quality improvement work.

If you can find the time, could you reassess the outstanding issues left after this period of code cleanup, and post them as a new issue, so this one can be closed. Thanks in advance!

I am not sure about the buffer and popping issues, I didn't really detect/understand them, these maybe need fixing with gstreamer's "queue"

fduncanh commented 1 year ago

I had to revert the change of default maxFPS ( the -fps setting) back to 30 because it caused problems on Raspberry Pi.

For the moment I added a suggestion in the README to try -fps 60.

fduncanh commented 1 year ago

@WinkelCode

It turned out that the non-syncing of client video with server audio in Audio-Only mode (eg. for Apple Music lyrics) was due to the gstreamer pipeline being in "sync=false" mode which ignores timestamps.

The new UxPlay 1.63 (in the "testing" branch on GitHub ) now gives an option "-sync" to switch on syncing in Audio-Only mode. (This introduces a delay) The option -ao x is replaced by -al x which tells an audio latency to the client, but doesn't seem to do much when the default 0.25 sec latency reported bu UxPlay is changed.

WinkelCode commented 1 year ago

@WinkelCode

It turned out that the non-syncing of client video with server audio in Audio-Only mode (eg. for Apple Music lyrics) was due to the gstreamer pipeline being in "sync=false" mode which ignores timestamps.

The new UxPlay 1.63 (in the "testing" branch on GitHub ) now gives an option "-sync" to switch on syncing in Audio-Only mode. (This introduces a delay) The option -ao x is replaced by -al x which tells an audio latency to the client, but doesn't seem to do much when the default 0.25 sec latency reported bu UxPlay is changed.

Unfortunately I can't provide feedback for the moment because my iPad broke, I'm currently trying to get my hands on a suitable replacement. I'll get around to testing and providing feedback once I do.

fduncanh commented 1 year ago

Sounds good! Please add future comments here or open a new issue if there is one

There is even more progress, released as v1.63

You comments were very helpful.

I now finally understand syncing in GStreamer

This make running on Raspberry Pi possible without hardware video decoding, so its interesting that your analysis on Windows fed back eventually to R Pi! (specifically the apparently "minor" issue of delayed lyrics in apple music opened up these issues...)

fduncanh commented 1 year ago

@WinkelCode

Also, when building with Bonjour, it would be useful to not have to install the Bonjour SDK (I extract the newest Bonjour version from the iTunes installer .exe which doesn't seem to come with the SDK?). I fixed this by getting the Bonjour SDK files from an alternative source (link in synergy-core's CI workflow) and putting the SDK files in the lib/ directory (rough workaround).

I found that the Bonjour SDK can be downloaded from Softpedia without going through the annoying registration process on the Apple site: See the revised UxPlay README Also the environment variable $BONJOUR_SDK_HOME can now point to an alternative location for it.

FDH2 / UxPlay

(Windows issues, some now fixed): Is it possible to set a buffer/delay? #169