srt-live-transmit high jitter

AlexandreLicinio commented 4 years ago

When srt-live-transmit is used as a gateway with udp as medium for sharing the jitter seen in a broadcast decoder (Ateme DR5000) is near 3ms.

./srt-live-transmit srt://:10199?mode=listener udp://239.127.0.1:11199 ./srt-live-transmit udp://239.127.0.1:11199 srt://:19911?mode=listener

When srt-live-transmit is used as a gateway without udp and straight srt to srt, the jitter is very low.

./srt-live-transmit srt://:10199?mode=listener srt://:19911?mode=listener

Any idea ?

maxsharabayko commented 4 years ago

Is it a packet inter-arrival jitter?

I think it is because of the buffering the app does. See this line. SRT-to-SRT transmission recovers timing, while udp to udp will suffer.

AlexandreLicinio commented 4 years ago

I guess yes, inter-arrival jitter, is it in the DR5000 contribution decoder from Ateme. It's RMS not peak to peak.

Ok, thanks. We are many requesting that srt-live-transmit will be able to transmit to many SRT receivers or with good udp. We cannot code in C but we use SRT with srt-live-transmit daily. Also, i don't get why sencore is reliable in udp > srt ou srt > udp while srt-live-transmit doesn't. The weirdest thing is if srt-live-transmit outputs to an udp multicast, as soon as you start services (ffmpeg, gstreamer, etc...) that are pulling that multicast the jitter goes high and ts discontinuity are massive. Whatever the env or the server. Keep in mind we are many using srt-live-transmit in production environnement and broadcast env. Thanks @maxsharabayko

maxsharabayko commented 4 years ago

@AlexandreLicinio srt-live-transmit was initially intended as a sample application to show how SRT API should be used and to provide some simple workflow for testing. But it turned to be actually applied in a production environment. This is something we would need to address, as more and more issues and requests are coming, that do not align well with the simple design of the app.

To avoid this jitter, I would suggest to try srt-test-live (use -DENABLE_TESTING=ON cmake build option) or srt-xtransmit route (link).

Both don't have that kind of buffering. Please tell if they resolve those jitter issues.

AlexandreLicinio commented 4 years ago

Max, with srt-test-live the jitter problem is sorted with -DENABLE_TESTING=ON, thanks. But ts discontinuity still there even with srt-xtransmit route when sending srt to or from udp.

maxsharabayko commented 4 years ago

A command-line option can potentially be added to srt-live-transmit to disable this buffering.

[x] Add a command-line option to srt-live-transmit to disable buffering.

@AlexandreLicinio Regarding TS discontinuity, it is probably related to missing packets, right? Although you seem to be streaming at around 5 Mbps, which is quite low, but please check the "UDP Performance" section of srt-live-transmit.md. Might be related to UDP buffer size. Additionally, you could grab statistics from SRT to check it is not dropping too-late packets.

Note the default receiver buffer latency of SRT is 120 ms, which is good for RTT up to 40 ms. If you have higher RTT on your link, you might want to increase the latency.

AlexandreLicinio commented 4 years ago

@AlexandreLicinio Regarding TS discontinuity, it is probably related to missing packets, right?

No, not related to missing packets. We already checked udp buffering. The udp outcoming packets are looking to be out of order or due to the jitter the decoder cannot put him in the right order.

maxsharabayko commented 4 years ago

Then it must be happening on UDP input. srt-live-transmit and other tools transmit packets in the order as they arrive on UDP.

AlexandreLicinio commented 4 years ago

In that case it must be the jitter causing the disorder, right ?

ethouris commented 4 years ago

In general, when the packet travels through the application, there could be 5 waypoints where the potential jitter might happen:

Entrance to the application (between the live stream sender and the input of srt-live-transmit).
In the application (between reading from input and sending to SRT)
In SRT
Between SRT endpoint and the other SRT endpoint
Between the SRT input at the receiver and the application.

SRT through the TSBPD mechanism ensures there's no jitter in waypoints 3 and 4, so only waypoints 1, 2 and 5 may apply.

Waypoint 5 is something like "the SRT does its best to give you the packet at the right time", but the SRT reader application may apply some delay between the moment when SRT reports the packet as ready to read and reading the packet by the application. This jitter is treated as something negligible and its significance declines with better system performance.

Waypoint 2 is currently the problem that can be potentially eliminated when we restore the enforced timestamp feature and use it by locking the timestamp at the moment when the packet arrives at the input. This way the timing of the packet could be defined by the times of arriving packets and transparently transferred up to the moment when the receiver application reads them.

Waypoint 1 is something that SRT can only "wash its hands of". To minimize the impact of it, the live streaming application should rather use the stdin input and give the SRT application packets at the right moments - although on typical POSIX system sending through UDP on local machines should make no difference.

AlexandreLicinio commented 4 years ago

Hi, we ran this test to reduce a number of variables on the 5 waypoints.

maxsharabayko commented 4 years ago

Hi @AlexandreLicinio Thanks for the results.

stransmit v1.2.3 did not have the buffering (code).
stransmit v 1.3.0 already has the buffering of up to 10 packets (code).

There was some reason to add that buffering. @rndi

Meanwhile, PR #1422 will provide a command-line option to set the size of the buffering. PR #1422 added -srctime:true command-line option to reduce jitter in SRT output with SRT/UDP/STDIN input. See this doc.

jpossetti commented 2 years ago

Hi, does anybody find a solution with srt-live-transmit on current release v1.4.4? I tried option -srctime:yes and -buffering:1 but glitches and TS discontinuity still happend (same TS discontinuity errors) with SRT to UDP. Only works as @AlexandreLicinio , only works fine with ./stransmit v1.2.3 but this script don't have srt stats that I need. Thanks!

maxsharabayko commented 2 years ago

Hi @jpossetti srt-live-transmit is a sample application intended to be simple by design to also serve as an example. It shows how to unify handling SRT and UDP using the epoll concept. However, SRT epoll API is not that efficient in handling UDP sockets (see #2303). srt-live-transmit uses SRT epoll API to handle both UDP and SRT sockets, thus it is affected by this issue.

NB! This issue is related to the SRT epoll API handling pure UDP sockets in addition to SRT sockets. The issue affects only SRT epoll usage with pure UDP sockets and does not affect SRT behavior itself.

There are two alternative applications not relying on SRT epoll API for UDP input and output:

srt-test-live (enabled with -DENABLE_TESTING=ON CMake build option) uses UDP socket in a blocking mode.
srt-xtransmit (link) by default uses non-blocking UDP socket and ::select for waiting. Both are better in terms of UDP performance compared to SRT epoll waiting used by srt-live-transmit.

These apps should be considered as an alternative for srt-live-transmit when doing these kinds of testing.

Haivision / srt

srt-live-transmit high jitter #1337