Reduce lag / delay during live streaming with HLS + fMP4 segments, transcoded by ffmpeg

[pcgeek86]

Description

I'm using ffmpeg to ingest an RTSP video/audio stream, and perform real-time encoding to HLS + fMP4 segments (chunks). The file chunks are hosted by the "Caddy" web server, which is reverse proxied by NGINX, which hosts a static HTML page with video.js, although this detail is relatively unimportant.

Here is what my ffmpeg transcoding command looks like, which is running at real-time:

~/Downloads/ffmpeg \
  -rtsp_transport tcp -i rtsp://username:password@ipaddress \
  -an \
  -filter:v fps=fps=15 \
  -preset veryfast \
  -f hls \
  -hls_init_time 0.3 \
  -hls_time 0.3 \
  -hls_playlist_type event \
  -hls_segment_type fmp4 \
  -b:v 1000k \
  -sc_threshold 0 \
  -g 8 \
   ~/Video/stream.m3u8

This generates an .m3u8 file that looks similar to the following, along with all of the corresponding .m4s data fragments (each of which is ~65-70 kilobytes):

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:1
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-MAP:URI="init.mp4"
#EXTINF:0.533333,
stream0.m4s
#EXTINF:0.533333,
stream1.m4s
#EXTINF:0.533333,
stream2.m4s
#EXTINF:0.533333,
stream3.m4s

As you can see, each chunk / segment / fragment is consistently 0.533333 seconds in length, so very, very short. This technique is the

Steps to reproduce

Explain in detail the exact steps necessary to reproduce the issue.

1. Run the ffmpeg command to initiate transcoding / muxing process
2. Host the files behind a web server (ie. NGINX, Caddy, Apache)
3. Use video.js to playback the live stream URL (ie. http://localhost:8080/stream.m3u8)
4. Notice that the video stream in browser is about 7 seconds behind the actual RTSP source

Results

Expected

My objective is to reduce delay in the live stream to under 1 second, between the RTSP source, and what's displayed in the browser.

Actual

Right now, the delay between the RTSP source and what's being displayed in the browser is around 7 seconds.

At the moment, I don't believe that any delay, beyond 0.5333333 seconds, is being caused by my ffmpeg transcoding / muxer process, or the web server processes that are simply hosting the files (Caddy + Nginx).

Error output

There are no errors. However, under the Network tab in the F12 Chrome Dev Tools, I observed that the video.js downloads about 5 chunks of video data, when it starts playing back the stream. My hypothesis is that the ~7 second lag is being caused by buffering several chunks, but playing the chunks from the beginning of the first chunk, instead of at the beginning of the last chunk. I would like to reduce the number of chunks that are buffered, and stream as "close to the source" as possible.

Additional Information

versions

videojs

what version of videojs does this occur with?

7.5.5

browsers

what browser are affected?

I have only tested with Chrome 79.

OSes

what platforms (operating systems and devices) are affected?

MacBook Pro 2018 w/ Intel Core i7 CPU MacOS Catalina 16GB RAM

plugins

are any videojs plugins being used on the page? If so, please list them below. No plugins, just the native player

Question: How do I ensure that the "playhead" of the video.js player is absolutely as far forward as it can go, to get as close to "real-time" as possible?

[welcome[bot]]

👋 Thanks for opening your first issue here! 👋

If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can. To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[Smith-Chris1]

One issue you'll face with HLS Dash, is players won't typically start playback until there are at least 3 segments buffered and cached, I believe you can define this, but it could cause buffering issues if your content is being requested too quickly. So in theory, using the .5 second chunks, you should be able to start playing 1.5 seconds after realtime.

Since you're doing a transcode, that will add some latency to this process, not much, but it could account for 1-2 seconds of latency. If you can do a -c copy in ffmpeg, this will help reduce that latency, but your bitrate would be whatever the rtsp stream is currently at. That may be too much for your viewers.

I haven't played with HLS-LL or Dash low latency, your segment sizes are closer to that length rather than normal HLS. I do know that you'll be more requests per second for HLS-LL, make sure your webserver is able to handle the amount of requests. Also ensure your web server is NOT caching the manifests, even a cache of 1 second will add to your latency.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TroyKomodo]

What was the outcome of this? @Smith-Chris1 , is there a way to get less then 7 seconds of delay with video js, as to not cache the playlist items and read them as they get added.

[gkatsev]

We are likely to start working on low latency streaming support soon.