Video stuttering problems

[kobuki]

When I use ustreamer, the stream on the client is unbearably stuttering. It causes buffering in mpv, and generally the stream playback stops after every 5-6 seconds for a few seconds, then continues. The playback is showing the same problems in mpv and in browsers as well.

Setup: Pi 4 4 GB, kernel 5.4.51-v7l+ #1327, Raspbian 10.4

Command line:

./ustreamer --device=/dev/video0 --format=YUYV --workers=4 --persistent --drop-same-frames=30 --host=0.0.0.0 --port=8080 --resolution=1280x720 --desired-fps=30 --encoder=omx

Version: 1.20, directly from git

I can provide logs if necessary. Using ffmpeg with the same capture device provides perfectly smooth streams, no choppyness or buffering (see ffmpeg command line in #23).

[mdevaev]

What capture device do you use?

Try to remove --drop-same-frames option.

[kobuki]

The device: Bus 002 Device 002: ID 1bcf:2c99 Sunplus Innovation Technology Inc. This exact model: https://www.aliexpress.com/item/32991659016.html

Removing --drop-same-frames helps in a browser but the frame rate is obviously not the requested 30 FPS, much slower. No intermittent stops. mpv on the other hand is still regularly buffering and stopping. No issues with ffmpeg or mjpeg_streamer.

mpv command line: mpv http://[IP]:8080/stream --no-correct-pts --fps=30

[mdevaev]

Try this: ./ustreamer --device=/dev/video0 --format=jpeg --host=0.0.0.0 --port=8080 --resolution=1280x720 --desired-fps=30.

[kobuki]

This device only supports YUYV, no built in encoding engine.

-- ERROR [25577.577 stream] -- Could not obtain the requested pixelformat=JPEG; driver gave us YUYV

[mdevaev]

If there are no problems with mjpg_streamer, we can try to simulate its behavior. Try ./ustreamer --device=/dev/video0 --format=yuyv --workers=4 --host=0.0.0.0 --port=8080 --resolution=1280x720 --desired-fps=30 --encoder=cpu. Also try --workers=1.

[kobuki]

Thanks, that command line seems to work. --workers=1 just makes it skip a lot of frames (cpu limit is hit). In Firefox, MPC-HC it works but mpv still intermittently stops. That's probably not ustreamer's fault, though with mjpeg_streamer and ffmpeg mpv works fine, so I'm not sure.

[kobuki]

More weirdness. ./ustreamer --device=/dev/video0 --format=yuyv --workers=1 --host=0.0.0.0 --port=8080 --resolution=1280x720 --desired-fps=30 --encoder=omx is working just as fine as with cpu now. I don't really get it. However, #23 still occurs with omx, sadly.

[mdevaev]

OMX probably doesn't have enough memory, and it requires some kernel configuration.

Add to /boot/config.txt:

gpu_mem=256
start_x=1

Add cma=128M to /boot/cmdline.

Reboot.

[kobuki]

I made all the changes, but nothing changed. No intermittent stops, but the "micro-stuttering" is present. It's like it's playing half-speed or a fraction. Tried again ffmpeg, works a lot better as of now.

EDIT: more like, about half the frames seem to be played, but with the correct speed. Hard to describe exactly.

[mdevaev]

Let's get in touch in the chat. If possible, I would like to view your device myself via tmate or ssh.

Write me to discord please, mdevaev#3927.

[kobuki]

Sure, thanks for the offer, but please let's arrange it tomorrow. I'm too tired now. I'll contact you on Discord tomorrow. And in any case, thanks for the excellent support and the attention you pay to make your tool better.

[mdevaev]

Okay, I'll be waiting.

[kobuki]

I've set up a tmate session as requested and pinged you on pikvm #chat.

[kobuki]

Through the support session, we concluded the following things:

• hw-assisted mjpeg compression can actually hit a hard limit with 1080p@30 (probably somewhere lower), causing lots of dropped frames, thus choppy playback
• lowering resolution and/or fps mostly fixes the above (with some slight choppyness), but it's not for all purposes (good for pikvm, tho)
• using cpu encoding works just fine, but a bit choppy, and the choppyness increases with resolution
• to reach an actual desired fps, a higher fps must be specified on the command line, eg. for fps of 30, one needs --desired-fps=40
• omx decoding has an issue that prevents the app from stopping and cleaning up properly - according to @mdevaev, it's caused by the following:

  OK, the hypothesis is that the mmap memory area of v4l2 from the driver cannot be copied to omx probably tricky arm assembly instructions are used for copying, and if we just compress jpeg with the processor, they are not used

With this card (and likely with other similar UVC cards with no hw compression), the smoothest playback can be achieved by using: ./ustreamer -d /dev/video0 -r 1280x720 --desired-fps 40 --workers=4 --encoder=omx

[mdevaev]

UPD: Hypothesis not confirmed

[mdevaev]

omx decoding has an issue that prevents the app from stopping and cleaning up properly

Fixed, details in #23.

[mdevaev]

OK, I made it so that the soft framdrop will be disabled if the hardware supports the specified desired FPS. I think that's the best thing I can do right now for this task.