Video Freezes on Raspberry Chromium

[freadZdead]

Hi @steveseguin ,

Back to the project with the "virtual" TV studio, made up off 3x Raspberry Pi 400s (slightly augmented Jessie lite in a kind of kiosk mode), each running chromium-browser in full screen, being the director and sending out feed from HDMI to USB capture cards (teleprompter) and receiving each a remote client (using vdo.ninja on chrome on their respective laptops), and going fullscreen out into a streaming atem mini extreme.

Performance wise, what we see is regular moments of complete video freeze on the Raspberry viewer side (so webcam content from the remote machines), while the audio appears uninterrupted and keeps going - video eventually unfreezes and is back in sync.

I tried:

• reducing the overall res of the pis to 720p (only works when hardware acceleration is off in raspi-config)
• using hardware acceleration (but sets the res to 1080p)
• using the method setting the ethernet adaptor to 100MBit as per your raspberry guide (but also sometimes to Gigabit Ethernet to compare)
• changing codecs (VP8 was autoselected)
• changing h264profile
• buffer (300 seemed to actually make it worse, 1000 seemed slightly better, but unacceptable latency in our situation)
• hint=motion on the remote side (as I wanted the webcam to favour a consistent framerate over detail)
• experimented with enabling chromium's hardware accelerations on the pis, but did not seem to have made a huge difference
• using vb=1000 on both sides to limit the bandwidth either side
• set the resolution of the HDMI to USB capture cards to 720p
• sharpen

Ultimately, this has not been fully resolved, and sometimes I suspect that it feels like a spike in content is causing the freeze - like a variable bit encoder...

• any tips?
• any way to set an constant bit rate to have a fully predictable load?
• If the CPU of the Pi is the problem (checked with htop that the RAM seems fine, and the 4 cores happily seem to all sit between 80 and 100%, I have never seen them all maxed out at the same time - but maybe I am kidding myself when I think that this might be a good sign, not sure how good the load balancing is) - is there a way/codec to lighten the load ("teleprompter" is mainly text, but also sometimes footage and audio)?
• And/or is there (apart from hint) a better way to configure the stream that avoids video freezes best, so that even if there is an adaptive loss of detail/resolution, the video keeps going alongside the audio?
• when using stats on the raspberry pi side, I am seeing consistent 0% packet loss, and only around 520kbps video bit rate, so with audio together around 600kbps, and a resolution of 960x540 - so it seems that either the vb flag is being ignored, or not more is needed.
• when using stats on the remote performer's side, I am seeing 640x360 only, and packet losses of up to 1% - but correct me, this is talking about the teleprompter coming to the performer, not the performer stream to the raspberry?

If it would lighten the load, I assume that we could afford using less compression, if the stress comes from difficult decoding rather than handling lots of Data bandwidth.

Here the links we are using per pi/virtual performer:

A) The Raspberry Pi assumes director (sending his own feed with the teleprompter content) and simultaneously views the remote performer:

https://vdo.ninja/?room=[roomid]&push=[directorFeedwhichcontainsteleprompter]&vdo=1&pw=[password]&np&as&webcam&view=[remoteperformer]&fullscreen&clean&vb=1000&chroma=000F&style=0&appbg=[animated gif for backdrop*]

B) The remote performer is viewer of the teleprompter/director, and sends their webcam:

https://vdo.ninja/?push=[remoteperformer]&autostart&webcam&room=[roomid]&pw=[password]&mini&vb=1000&appbg=[animated gif for backdrop*]

*note, because of the scaling, I had to make this fill a 16:9 frame, and while it is a highly compressed gif, it still comes to ca 700kb

Thanks for letting me know what can be done :)! with the lowered res and lowered bitrate I had hoped to get a consistent if slightly lower quality image, but the freezes are killing it a bit at the moment...

Needless to say - we are grateful that the audio at least seem to continue during the video freezes, as full freezes would be even less ideal...

Thanks so much for sharing your wisdom, and your continued development of such a cool platform :)!

EDIT: I have also read the info around freezes, but none of that has helped thus far.

Interestingly, I noticed an interesting pattern emerging - when there is little movement (i.e. nodding and importantly - silence), it seems to stay fairly steady and uninterrupted, and as the show is designed, there is definitely very clear "turn taking" between each remote performer - but as soon as we switch (in the atem) to one of the performers, and said performer starts to read from there teleprompter, the freeze occurs (for up to 8 seconds, just the video), but then, when there is consistent action (slight movement of the actors, speaking), there is less likelihood of the freezes. It is as if the change from silence to audio causes a spike in audio to then freeze the video...

Is there a non-intrusive way to log how the stream transmission is going for forensics after the fact, ideally from a third browser window on a separate computer (maybe as a director)? Would be good to be able to debug what happens exactly at the moments of the freeze, i.e. being able to tell the difference between encoding/transmission/decoding issues?

Cheers, Freddy

[LagaV]

@freadZdead Did you make any progress on this? I'm currently seeing the same challenge (also a teleprompter) usecase, but even without sending the raspberry webcam, the stream is significantly delayed (upto 10 seconds in general and also jumping to 40 seconds, then recovering to nearly live speed).

[steveseguin]

I have https://raspberry.ninja/ , which should allow for 1080p30 video publishing on a Raspberry Pi to VDO.Ninja

It doesn't use the browser, but rather a Python script. This frees up a lot of CPU and uses the hardware accelerator to encode the video.

Publishing video via a Raspberry Pi's Chromium browser isn't something I can really offer good results from. You can try adding &codec=h264 to the viewer side, to see if that triggers the hardware h264 in the pi, but changes are it will be using the software encoder.

Lowering the capture resolution to 360p (&q=2) might help, and using a webcam that has raw yuy2 output rather than h264/mjpeg output might free up additional CPU cycles. The CSI camera should have raw yuy2 output I think.

You can further try to force a high resolution and low frame with &mfr=5 on the rpi side, or play around with different values.

You can also maybe try &contenthint=detail or &content=text to see if that helps to get the pi to prioritize resolution over frame rate.

Another option is to add &meshcast to the RPI's sender link, which will send the data to a server. This would avoid the situation where you had multiple viewers of the RPI, slowing it down. Using meshcast ensures the video is only sent to one viewer -- a broadcast server, and that handles restreaming to other viewers.

If you are seeing huge latency, you can also try having Firefox on your PC to PLAY BACK the video; FIrefox might not increase the buffer as much as Chrome does. Lowering the capture resolution on the pi might also help.

Kind regards, steve

[LagaV]

I got the delay down to sub-second value by re-enabling the hardware graphics acceleration and disabling the display mirror flip option and reducing the resolution.

But I'll transfer this overall task (teleprompter) over to a cheap Lenovo ThinkCentre T700 on Windows.

[steveseguin]

Thank you for the that feedback