[ENHANCEMENT] Better/Smoother playback from rtc stream

nurdism / neko

A self hosted virtual browser (rabb.it clone) that runs in docker.

Apache License 2.0

2.04k stars 184 forks source link

[ENHANCEMENT] Better/Smoother playback from rtc stream #48

Open stevelord opened 4 years ago

stevelord commented 4 years ago

Describe the bug While playing videos, there's a general jerkiness every few seconds.

To Reproduce Steps to reproduce the behavior:

Configure a CPU-optimized Digital Ocean droplet with 8Gb of RAM in your closests datacentre
Install Neko as per instructions
Log in, go to this trailer, play it, full screen it, full screen your browser.
Note the jerking every few seconds, with jerking and tearing particularly noticeable in camera pans

Expected behavior Video playback as smooth as a baby's bum

Desktop (please complete the following information):

OS: MacOS
Browser: Chrome, Safari
Version FF 74.0, Chrome 80.0.3987.149

Additional context I'm not sure if this is a problem with X11, or X11VNC but I've tried on different systems with different browser combinations and it's consistent. Happy to look into it if someone can give me some pointers.

CPU load is fine, RAM is fine. There could be a sync and framerate issue or something can't keep up.

nurdism commented 4 years ago

I've noticed some stuttering myself occasionally. I believe its the encoding/gstreamer that is lagging behind when attempting to get low latency rendering.

You can see here where the settings are passed to gst https://github.com/nurdism/neko/blob/master/server/internal/gst/gst.go#L87 I've been able to get good smooth video but at the cost of the latency being way higher. You more than welcome to mess with it and see if you can get something better going.

stevelord commented 4 years ago

Do you know what settings you used for smooth video? I'm trying to build at the moment but I'm finding the whole VS/docker setup a bit difficult as I'm on a Mac. Working around it, but it's slow.

GigaFyde commented 4 years ago

Cannot exactly reproduce on my dedi.

What resolution settings in neko are you using?

GigaFyde commented 4 years ago

Also, neko is incredibly reliant on the udp stream that carries the video data to your client, if that connection experiences hiccups the stream will stutter.

stevelord commented 4 years ago

@GigaFyde it's all the default settings from the build.

I did find something interesting last night. I built a derived docker container launching VLC instead of Firefox or Chromium, mounted a volume with the media and it was much better. Not perfect, but better.

nurdism commented 4 years ago

oh if that's the case, that might be a limitation of running a browser in a docker container.

stevelord commented 4 years ago

It appears the pipeline is quite complex if you think about it, we have:

A browser that has to decode video, render images, play sound and draw it all in X11
A Pulseaudio setup that has to handle sound
An X11VNC instance that has to take the images and convert to VNC (I think)
A Gstreamer pipeline that has to reencode everything
A Nodejs setup that then has to stream the output to multiple browsers

I think I'm wrong about some of the above, I've only seen the parts individually and haven't spent enough time digging to be sure what's used when. I wonder if there's a more optimal way of handling this. I'm going to try looking at the IO stats next time I run Neko and see if the content is being written to disk, read from disk repeatedly by different stages.

nurdism commented 4 years ago

You almost have it right, the current system is :

Browser does its thing
X11 via xserver-xorg-video-dummy
Pulseaudio sinks
Neko (using golang) uses Gstreamer pipelines and captures raw X11/Pulseaudio output and encodes, then sends them to webrtc clients.

The only improvements I can see is to optimize the Gstreamer pipelines or find a better webrtc lib/alternative lib, the client is 100% static html/js once its been generated so no performance impact there.

zaeek commented 4 years ago

Hello, can u rewrite the code in js/node , it's hard for me to understand the code in vue.js. Thanks :dagger:

stevelord commented 4 years ago

@zaeek that might be best served as at least as a separate issue, if not also a pull request.

stevelord commented 4 years ago

As an aside, I ran another session last weekend and had horrible dropouts. It looks like some of this may be coming from latency on my broadband connection introduced through quarantine and furlough. I've noticed this isn't happening with Plex, but plex is streaming video, where as Neko will stream individual frame state.

GigaFyde commented 4 years ago

That is indeed caused by the differences in how both serve the video feed,