Reduce video stream lag

[somik123]

Currently, the lag is between 0.8s and 1.20s using ustreamer.

I recommend swapping it out with mjpg-streamer. As the HDMI-to-USB dongle is providing video in mjpeg format, mjpg streamer can stream it with notably lower lag. From my tests, i would say it is bellow 0.2s. Can you please check?

Installation steps for standalone server:

sudo apt update
sudo apt upgrade

sudo apt-get install build-essential libjpeg8-dev imagemagick libv4l-dev cmake git

mkdir ~/mjpg-streamer
cd ~/mjpg-streamer

git clone https://github.com/jacksonliam/mjpg-streamer.git
cd mjpg-streamer/mjpg-streamer-experimental

make
sudo make install

Start the video stream server by running: /usr/local/bin/mjpg_streamer -i "input_uvc.so -f 30 -r 1920x1080" -o "output_http.so -w /usr/local/share/mjpg-streamer/www"

Yes, I was running it at 1080p at 30 fps. Didn't see any lag between my local and browser stream.

The stream runs on http://raspberrypi:8080/?action=stream

I played around with it for a bit and it seems to perform MUCH faster then ustream with nginx proxy.

[mtlynch]

It sounds like you're comparing nginx+ustreamer to mjpg-streamer directly. Would it be possible to do an apples-to-apples comparison to make sure nginx isn't what's introducing the latency?

[mtlynch]

To add context, my reluctance to abandon ustreamer is that I'm not certain TinyPilot will stick with the current HDMI dongle that does onboard MJPEG streaming. If we switch to a chip like the tc358743, uStreamer lets us use the Pi's onboard GPU to accelerate video encoding.

[somik123]

It sounds like you're comparing nginx+ustreamer to mjpg-streamer directly. Would it be possible to do an apples-to-apples comparison to make sure nginx isn't what's introducing the latency?

Ya, i was actually looking at the nginx configuration.

~Btw, anyway to stream ustreamer output directly to check? From what I can tell, it is only streaming on internal port 8001. Anyway to make it appear on external port?~

Used nginx to stream the port instead of proxing it. Still same delay.

I'll test with mjpg streamer with nginx proxy and see how it goes.

[somik123]

Compiled both ustreamer (https://github.com/pikvm/ustreamer) and mjpg-streamer (https://github.com/jacksonliam/mjpg-streamer) on same Pi, ran through same nginx server.

On same hardware, mjpg streamer works better with noticeably lower lags. HDMI disconnection and re-connection works just as well, atleast with the recommended HDMI to USB dongle.

I ran ustreamer with: ./ustreamer -d /dev/video0 -r "1920x1080" -f 30 --host=0.0.0.0 --port=9080 Note that dropping the frames to 10 did not help, but lowering the resolution seem to make it lag slightly less.

And mjpg-streamer with: /usr/local/bin/mjpg_streamer -i "input_uvc.so -f 30 -r 1920x1080" -o "output_http.so -w /usr/local/share/mjpg-streamer/www"

[mdevaev]

Can I find out exactly how you measured the signal delay?

[somik123]

Run a stopwatch, take photo of both screen with phone camera.

With Pi on wireless, monitoring PC running on wired.

With ustreamer:

With mjpg streamer:

.

.

.

.

This is running on all wired connections:

With ustreamer:

With mjpg streamer:

[somik123]

Played around with the ustreamer settings, but still significant latency, both with hardware and software decoding and changing the input formats.

Possible to add a option to switch between the two streams from GUI or when installing or after installation?

[mdevaev]

1) I just took measurements with dongle and tc358743 using a similar method and see that the delay is the same.

2) You're running ustreamer incorrectly. Rather, you are selecting the wrong options to ensure that you match the same criteria.

[mtlynch]

@somik123 - this is the command TinyPilot uses to run uStreamer (from /lib/systemd/system/ustreamer.service):

./ustreamer \
  --host 127.0.0.1 \
  --port 8001 \
  --encoder hw \
  --format jpeg \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent

Also, superficial, but I like this stopwatch as it offers a clean fullscreen view.

[somik123]

@somik123 - this is the command TinyPilot uses to run uStreamer (from /lib/systemd/system/ustreamer.service):

./ustreamer \
  --host 127.0.0.1 \
  --port 8001 \
  --encoder hw \
  --format jpeg \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent

Ya, I already found it and copy pasted that for the second set of photos. The only change i made is change host to 0.0.0.0 as i wanted to directly access

Also tried with:

/opt/ustreamer/ustreamer \
  --host 0.0.0.0 \
  --port 8001 \
  --encoder HW \
  --format JPEG \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent \
  --quality 100 \
  --io-method USERPTR

/opt/ustreamer/ustreamer \
  --host 0.0.0.0 \
  --port 8001 \
  --encoder HW \
  --format JPEG \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent \
  --quality 100 

/opt/ustreamer/ustreamer \
  --host 0.0.0.0 \
  --port 8001 \
  --encoder CPU \
  --format JPEG \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent \
  --quality 100 

Result delays were always higher then with fmpg streamer.

[mdevaev]

@mtlynch remove --workers 4: it's not a necessary for hw encoder.

[somik123]

@mtlynch remove --workers 4: it's not a necessary for hw encoder.

@mdevaev Also tried:

/opt/ustreamer/ustreamer \
  --host 0.0.0.0 \
  --port 8001 \
  --encoder HW \
  --format JPEG \
  --desired-fps 10 \
  --resolution 1920x1080 \
  --workers 1 \
  --buffers 1 \
  --drop-same-frames 30 \
  --persistent

Buffers and workers cant be set to 0, but their minimum 1. Still same...

[mtlynch]

@mdevaev - Thanks! Fixed in https://github.com/mtlynch/ansible-role-tinypilot/pull/34

@somik123 - Over a wired connection, they're performing about equally, right?

I'm reluctant to switch to mjpeg-streamer because it's not very actively maintained. It's got 15 stale PRs right now. uStreamer is well-maintained and documented.

I'd consider a switch to gstreamer since it's so large and actively maintained, but I'd need to spend a lot more time learning to use it and figuring out whether it matches TinyPilot's use case.

[somik123]

@somik123 - Over a wired connection, they're performing about equally, right?

I'm reluctant to switch to mjpeg-streamer because it's not very actively maintained. It's got 15 stale PRs right now. uStreamer is well-maintained and documented.

I'd consider a switch to gstreamer since it's so large and actively maintained, but I'd need to spend a lot more time learning to use it and figuring out whether it matches TinyPilot's use case.

Yes, that's what i found strange. Over wired, the delay is noticeable, but is almost the same as mjpg-streamer.

When it comes to wireless, the delay shoots up to 1000ms for ustreamer while mjpg-streamer stays at around 200ms. Not sure what is causing it...

[mdevaev]

1) The lower the fps, the greater the delay. I think it's obvious.

2) All tests were incorrect. First you used the CPU encoder, which gives a significant delay on the RPi for large frames, and then you took options from the system service. You can't use the option --drop-same-frames without a full understanding of what it does and how it is handled by different browsers. This leads to huge delays. For Chrome/Blink, you need to use the URI parameter advance_headers=1 in combination with this option. For Safari/WebKit - dual_final_frames=1.

If you use the correct combination of options, the delay will be around 100ms. If you configure ustremer so that its behavior matches the mjpg-streamer, you will get equal or less latency.

[somik123]

@mdevaev

1. The lower the fps, the greater the delay. I think it's obvious.

Yes.

2. All tests were incorrect. First you used the CPU encoder, which gives a significant delay on the RPi for large frames, and then you took options from the system service. You can't use the option --drop-same-frames without a full understanding of what it does and how it is handled by different browsers. This leads to huge delays. For Chrome/Blink, you need to use the URI parameter advance_headers=1 in combination with this option. For Safari/WebKit - dual_final_frames=1.

I did miss out the URI parameter. I only put in http://192.168.1.155:8001/stream so I'll try with http://192.168.1.155:8001/stream?advance_headers=1 as I'm using chrome.

If you use the correct combination of options, the delay will be around 100ms. If you configure ustremer so that its behavior matches the mjpg-streamer, you will get equal or less latency.

That's the objective. Since mjpg-streamer can do it, I wanted to find out what set of options will allow ustreamer to do the same over WiFi as ustreamer lags significantly over WiFi.

[mdevaev]

http://192.168.1.155:8001/?action=stream&advance_headers=1 as I'm using chrome.

For ustreamer you need http://192.168.1.155:8001/stream?advance_headers=1.

That's the objective.

Try ustreamer -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30 vs mjpg_streamer -i "input_uvc.so -f 30 -r 1920x1080" -o "output_http.so -p 8081".

Since mjpg-streamer can do it

To be honest, I don't know what the problem is and I don't see mjpg-streamer transmitting data over wifi as successfully. I just did a test and made sure that both programs have the same delay of ~1s on my old laptop connected via wifi. When the cable is connected, the delay of both streamers becomes ~100ms.

[somik123]

Try ustreamer -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30 vs mjpg_streamer -i "input_uvc.so -f 30 -r 1920x1080" -o "output_http.so -p 8081".

Ok, I ran both over 5Ghz WiFi as usual, and I must ask, what source of sorcery command was that? Lag came right down to 110ms, over WiFi!

Using ustreamer:

.

.

Using fmpg streamer:

[somik123]

The logs generated, in case you need:

pi@raspberrypi:~ $ /opt/ustreamer/ustreamer -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30
-- INFO  [775.533      main] -- Installing SIGINT handler ...
-- INFO  [775.534      main] -- Installing SIGTERM handler ...
-- INFO  [775.534      main] -- Ignoring SIGPIPE ...
-- INFO  [775.534      main] -- Using internal blank placeholder
-- INFO  [775.534      main] -- Listening HTTP on [::]:8001
-- INFO  [775.534    stream] -- Using V4L2 device: /dev/video0
-- INFO  [775.534    stream] -- Using desired FPS: 30
================================================================================
-- INFO  [775.534      http] -- Starting HTTP eventloop ...
-- INFO  [775.771    stream] -- Device fd=8 opened
-- INFO  [775.771    stream] -- Using input channel: 0
-- INFO  [775.771    stream] -- Using TV standard: DEFAULT
-- INFO  [775.772    stream] -- Using resolution: 1920x1080
-- INFO  [775.772    stream] -- Using pixelformat: JPEG
-- INFO  [775.774    stream] -- Using HW FPS: 30
-- INFO  [775.774    stream] -- Using IO method: MMAP
-- INFO  [775.814    stream] -- Requested 5 HW buffers, got 5
-- INFO  [775.838    stream] -- Capturing started
-- INFO  [775.838    stream] -- Switching to HW encoder because the input format is (M)JPEG
-- ERROR [775.838    stream] -- Device does not support setting of HW encoding quality parameters
-- INFO  [775.839    stream] -- Using JPEG quality: encoder default
-- INFO  [775.839    stream] -- Creating pool with 1 workers ...
-- INFO  [775.839    stream] -- Capturing ...
-- INFO  [781.571      http] -- HTTP: Registered client: [::ffff:192.168.1.100]:36958, id=38f5346d-58a3-455c-bc05-d59d5ed42db0; clients now: 1
^C-- INFO  [920.358      main] -- ===== Stopping by SIGINT =====
-- INFO  [920.358      http] -- HTTP eventloop stopped
-- INFO  [920.361    stream] -- Destroying workers pool ...
-- INFO  [920.364    stream] -- Capturing stopped
-- INFO  [920.376    stream] -- Device fd=8 closed
-- INFO  [920.377      main] -- Bye-bye

Stream accessed by Chrome on URL: http://192.168.1.155:8001/stream?advance_headers=1

[mdevaev]

Well, it looks like the problem is solved. There is a fundamental bug in Chrome that causes the frame to be thrown only when it gets the headers of the next one. There are many other subtleties. Ustreamer is like a small nginx, it requires fine tuning to work well. By the way you can pull fresh ustreamer and try to add option --tcp-nodelay. This may reduce the delay a little more.

I don't want to brag, but before writing ustreamer I carefully read all the sources of all other streamers. You can even find my big patch in mjpg_streamer's upstream :)

[somik123]

Well, it looks like the problem is solved. There is a fundamental bug in Chrome that causes the frame to be thrown only when it gets the headers of the next one. There are many other subtleties. Ustreamer is like a small nginx, it requires fine tuning to work well. By the way you can pull fresh ustreamer and try to add option --tcp-nodelay. This may reduce the delay a little more.

Pulled a fresh copy and ran the test again. Adding --tcp-nodelay seems to make the lag slightly better, but still around the 110 to 150ms, which in my opinion is negligible.

@mtlynch Possible to update the installer with the new command? Delay reduces from 1000ms to 110ms over WiFi!

Replace:

ExecStart=/opt/ustreamer/ustreamer \
  --host 127.0.0.1 \
  --port 8001 \
  --encoder hw \
  --format jpeg \
  --desired-fps 30 \
  --resolution 1920x1080 \
  --workers 4 \
  --drop-same-frames 30 \
  --persistent \

With

ExecStart=/opt/ustreamer/ustreamer \
  -m jpeg \
  -r 1920x1080 \
  -q 100 \
  -s 127.0.0.1 \
  -p 8001 \
  -f 30 \
  --persistent 

.

.

.

I ran /home/pi/ustreamer/ustreamer -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30 --tcp-nodelay

[somik123]

@mtlynch Do note that your repo of mtlynch.ustreamer is not up to date and does not contain the new revision for "--tcp-nodelay" command.

[somik123]

Seems like adding "--tcp-nodelay" increases the lag once you pass the whole thing through nginx again for tinypilot... So my final config will look like this:

File: /lib/systemd/system/ustreamer.service

[Unit]
Description=uStreamer - Lightweight, optimized video encoder
After=syslog.target network.target

[Service]
Type=simple
User=ustreamer
WorkingDirectory=/opt/ustreamer
ExecStart=/opt/ustreamer/ustreamer \
  -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30 \
  --persistent \

Restart=on-abort

[Install]
WantedBy=multi-user.target

[mtlynch]

@somik123 - which of those flag changes affects the latency? A lot of them are superficial like changing from the short flag name to the long flag name. -s :: just changes ustreamer to listen on all interfaces rather than on localhost (--host 127.0.0.1), which shouldn't affect latency

[somik123]

@somik123 - which of those flag changes affects the latency? A lot of them are superficial like changing from the short flag name to the long flag name. -s :: just changes ustreamer to listen on all interfaces rather than on localhost (--host 127.0.0.1), which shouldn't affect latency

From what i gathered, removing these two seemed to be the key in reducing latency.

  --workers 4 \
  --drop-same-frames 30 \

As you said, --format jpeg overrides --encoder hw and ignores -q 100 so keeping or removing them should not matter.

This is the current setting and even with nginx proxy, the latency is well bellow 200ms over 5Ghz WiFi compared to 1000ms from before.

/opt/ustreamer/ustreamer \
  -m jpeg -r 1920x1080 -q 100 -s :: -p 8001 -f 30 \
  --persistent 

[mtlynch]

Thanks for measuring! I removed those parameters in the latest ansible-role-tinypilot:

• https://github.com/mtlynch/ansible-role-tinypilot/pull/34
• https://github.com/mtlynch/ansible-role-tinypilot/pull/36