Latency drifts

[DaveStein]

Description

When I watch a livestream, it's about 18-20 seconds behind realtime, which is normal. However, I can frequently see it drift so as a stream goes on for longer periods of time, it can go beyond one minute behind. Sometimes refreshing the player fixes it, other times it doesn't. The same issue was not seen on HLS.js nor Shaka.

https://jsfiddle.net/Dave_Stein/gf5Ler8s here is a fiddle of just videojs. https://eastus-1-cdn-prod.azureedge.net/5adad37c-adb2-47fe-afe3-400f93dc334b/28673a94-d50f-4236-bf49-af76589285cf.ism/manifest(format=m3u8-aapl,audio-only=false) is the manifest that I saw this last. Obviously it's not live anymore but it's possible there's something about discontinuity handling that is different in videojs vs other players.

Steps to reproduce

Explain in detail the exact steps necessary to reproduce the issue.

1. Have someone stream RTMP for > 45 minutes with https://onlineclock.net/ time on their screen
2. Occasionally check in on video playing in videojs
3. Have https://onlineclock.net/ open on your machine.
4. Compare the lag, see it change over time

Results

Expected

A consistent latency / glass to glass time.

Actual

The latency increased over time sometimes, would randomly fix itself other times

Error output

If there are any errors at all, please include them here.

Additional Information

Please include any additional information necessary here. Including the following:

versions

videojs

what version of videojs does this occur with? Several, but the fiddle uses 7.10.2

browsers

what browser are affected? At least Chrome & Safari.

OSes

what platforms (operating systems and devices) are affected? macOS & Windows

plugins

are any videojs plugins being used on the page? If so, please list them below. None in fiddle

[gkatsev]

I think hls.js and shaka do a better job at keeping in-sync with live. I think we mostly just play it and hope for the best. I'm a bit surprised that you say that you still see it in Safari because by default we use native playback and I don't see it being overridden.

We're currently working on getting LL-HLS in, so, this may be something we'd want to look at as part of that.

[DaveStein]

@gkatsev Is there a timeline on LL-HLS?

[gkatsev]

Only thing I can promise is that we're working on it right now. We want it out as soon as possible but realistically, it'll be out when it's ready.

[DaveStein]

@gkatsev understandable answer :) But do you think that would be within one month or further out? Would help me decide some work over here.

[gkatsev]

Unlikely to be done within the next month.

[DaveStein]

Thanks @gkatsev. I will try and double check the Safari report. I could be mistaken there.

[DispatchCommit]

This isn't perfect, but it's been my workaround for users for a few months now... I just have an option in my player settings for users called "keep live" and as long as their internet can keep up, it gets the job done. Definitely feels like a crude method though.

setTimeout(() => {
            // Do not jump ahead if user has paused the player
            if ( this.player.paused() ) return;

            console.log( 'Attempting to jump ahead and catch up to live edge now...' );
            this.player.liveTracker.seekToLiveEdge();
          }, 8 * 1000 );

I think video.js tries to "play every segment" in an HLS stream, and occasionally your PC might lag and drop a few frames, or internet slows and you buffer for a short segment, and overtime it adds up.

I've tried to use the liveedgechange event to trigger a seekToLiveEdge(), but liveedgechange doesn't seem to fire reliably and thus a setTimeout proved to be the most reliable.

Another attempt I had was to increase playback rate when falling behind live, but there's no event fired once you reach the live edge again (at least not on PC, on mobile android it seems to work). But this also proved to be unreliable.

[gkatsev]

I think video.js tries to "play every segment" in an HLS stream

That's exactly right. We'd probably want a config option that's like "keep at live edge at all costs" or something. The liveTracker can detect some drift now, so, I guess we just want it to seek to live when it detects drift. Increasing the playbackRate probably works a bit better from the internals as we have more hooks into what's happening. Though, maybe listening to the segment-metadata track to know where you're at could help with that.

[DispatchCommit]

Increasing playback rate is something I got as an idea after hearing lag on discord voice chats where if your network lags for a second, once it reconnects you quickly get a burst of what you missed (up to a point).

I think the video equivalent would be to play at 1.5-2.0x speed some X number of segment files. If you are behind more than X segments, just skip to the last X segments, download and play at a faster rate.

The only issue I ran into trying to do this from the dev side was that when fast forwarding, I am not sure how to detect when the play head has caught up to the most recent segment. So if I detect we've fallen behind, and then set the playback rate to 2x speed, playback continues at 2x speed all the way to the live edge, and then will buffer everytime it plays through a segment file.

I do live HLS streams, so my segments are 2 seconds long, so that becomes very choppy. If I could just detect when we've caught back up to live, I would have a decent work around for the short term. I expected liveedgechanged to fire when we hit the live edge, but won't fire when you are at a higher playback right.

is there any way to detect this right now? or would I have to wait for some internet development to be done to expose some more control at a higher level?

[DispatchCommit]

We'd probably want a config option that's like "keep at live edge at all costs" or something

Another option that would be nice would be the ability to set a threshold value (either in seconds, or in segment counts) similar to liveTolerance that would fire a seekToLiveEdge() if the threshold is exceeded. But only do this if current playback is within something like 2 * liveTolerance threshold so that way someone can still rewind through the DVR in a livestream.

[gkatsev]

Theoretically, we'd want a whole host of options for tweaking how latency is handled because it's not possible to have a single set of things work for everything perfectly. As for playback rate, I imagine more like setting it to 1.1 when you're a tiny bit behind to catch up, so that it isn't really noticeable to the user. Theoretically, you could also set playback speed to 0.9 or something if you end up being too far forward too.

[DispatchCommit]

Sounds great, I love the plan to expose and allow modifications to deeper player properties. I'll keep an eye out for discussions and PR's related. Would these changes need to be at the video.js level or deeper at like the http-streaming / vhs library level?

[gkatsev]

We're most likely only going to focus on VHS. We try and not mess around with native playback much if we can help it. Though, I guess we could put some of this into the liveTracker.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gabek]

I've been working on a little addon that attempts to manage, and compensate for latency in the player when possible. I wouldn't mind packaging this up and making it an installable videojs plugin, but admittedly I'm using a ton of .tech({DON'T USE THIS IN A PLUGIN! I KNOW WHERE YOU LIVE!}) vhs calls, and I have no idea how to change this so it can be a distributed piece of code without the vjs people showing up at my door.

[gkatsev]

In Video.js 6, we've loosened things up a bit. If you call player.tech() directly it'll work, but it'll log a warning since interacting with the tech directly is sort of an advanced use-case, and we still want to slightly discourage folks from using it if they can. Your plugin sounds like a reasonable thing to be using the tech directly, since you're interacting with VHS. You can silence the warning by calling it via player.tech(true).

[gabek]

I just created a new repo for the current state of some approaches to minimize latency when using the videojs http-streaming tech. It's not something I could see ever rolling upstream into videojs or vhs tech itself since at its core it is purposefully breaking the 3 segment buffer best practices spec and tries to get you to a "more live than what videojs says is live".

It's shipping as an experimental feature in Owncast 0.0.12. It has many flaws, but It'd be cool to work with a few people on tightening it up and offering it to anybody who wants it. You can try it out at https://watch.owncast.online/ and maybe it'll work for your playback experience.

https://github.com/owncast/vhs-latency-compensator

Happy to discuss more about how it works and what needs to be improved.

[gabek]

@DispatchCommit @DaveStein I just wanted to ping you two quick to see if you'd be interested in working with me on the above "plugin" to help with live latency. I'm using it with Owncast, and while it has issues, it has promise. I'd love to get some more eyeballs and ideas involved. https://github.com/owncast/vhs-latency-compensator