Virtual studio setup questions

[freadZdead]

Hi Steve,

Amazing projects, both vdo.ninja, electron capture, and the RPI integration of vdo.ninja, great stuff! I have been in a creative development working with vdo.ninja to get three remote AV participants into a local video mix, as well as provide them with an individual return AV for individual instructions, and then stream the mix out. I have achieved this in principle on MacOS, using a variety of tools Incl. OBS, Electron capture, and NDI/Syphon-to-cam drivers for the returns, and using 3 individual vdo.ninja rooms, but ultimately, it was a bit of a clunky solution that asked too much of the host, leading to a bottleneck in the final encoding/streaming.

Long story short, I saw your video on Jetson and RPI, and wondered if (with current implementation) this might be a possible solution:

• Use the Mac to output the three return streams via HDMI
• Use 3x RPI 400 (looking at the integrated keyboard version here) plus a HDMI to USB dongle on each, connected via ethernet, to A) capture and send the Mac's return stream outputs to one of the three individual rooms each, and B) simultaneously receive the remote AV feed and output via one of the RPI's own MicroHDMI ports, and take it into a streaming ATEM Video switcher (I think I have read that there is no full-screen output yet, but even using a crop of a chromium window would work for me I'd think, as I could set the res to 1080p, and only use a 720p windowed image).

I should say that I am not chasing crazy quality, I would be OK with 720p overall, and would even consider only returning 360p to the remote participants, but ideally reliable audio sync would be a thing I'd be chasing in both directions...

Would you be so kind as to let me know the caveats as they stand before I embark too far onto this journey (I am of course willing to persist and figure stuff out as I go, but if there are general, insurmountable road blocks, it would be amazing to know now before investing the time, if that makes sense)?

Thank you so much,

Freddy

[steveseguin]

Hey @freadZdead ,

Reliable audio sync is the one thing I'm struggling with right now with raspberry-ninja. 1080p is possible with an HDMI to USB dongle, and 720p should be overall pretty fine, but for some reason audio is struggling a bit.

There probably is a simple tweak to the code to get it working, but I haven't had time to grind out a solution.

A single Raspberry Pi 400 can probably multicast a single stream to multiple viewers at 1080p30, or 720p30 if there is heavier action in the video, when using Raspberry-Ninja (audio issues aside).

You can use Chromium as an alternative to Raspberry-Ninja, but it won't be using a hardware-encoder probably for the video. As a result, 360p video (maybe 540p) is about the limit for output. It will result in a pretty smooth stream and audio sync shouldn't be as much of an issue, or an issue at all. There probably are some tweaks to apply to the RPi OS to get things even smoother, and I mention a few tweaks in the Raspberry-Ninja project. (helps with packet loss, etc).

As for using VDO.Ninja + Raspberry pi as a source for a ATEM, many users currently do that, and it works fine with just a Chromium browser in something like a kiosk mode. 4 Raspberry Pis into an ATEM should allow for low latency video with synced audio, although using Ethernet is always recommended for best results. I don't know what codecs may be supported to get hardware accelerated video in this setup, but I suspect just h264 is the only possible option there. 720p seems reasonably obtainable even with software decoding on a RPI 400 , but I don't have one to test with.

The discord community might have some folks there with more feedback for you.

discord.vdo.ninja

[freadZdead]

Hi @steveseguin ,

Big EDIT: With time and research, I have been able to resolve most of the issues I mentioned below, which just goes to show, right? I have hence condensed the items I still do need help with here, and made the historic post below in reply format - just in case somebody else stumbles upon this and needs help figuring it out, I can try and assist.

Thank you for your response, and just clarifying - I am not after multi-casting - each pie would deal with one single outgoing AV stream to one single (remote) recipient (as director of the room, using USB HDMI dongle, which receives its input from another computer), and one single incoming stream (the solo link of the remote participant, full screen, and out via its MicroHDMI output as one of several sources to a streaming ATEM switcher) - so we are actually talking 3 rooms (one per pie), with one director and one participant each.

I have since done quite a bit of tinkering, with the following results - I used your image as a starting point and added limited Xorg and gui and chromium, and seem to see fairly OK sync and latency at 720p in both direction - which is an acceptable quality for me. I also experimented with the kiosk image that you recommended, but since we have control over the devices, that was just a little bit too restrictive. The HDMI USB Capture was btw just powering off based on standard power saving USB settings in linux, I disable this on boot-up for all connected devices. And after some tinkering, I have the best of both worlds, with both the Window Manager and Chromium in full-screen, as well as an optional VNC server running in the background.

What is left (and I cannot seem to change that, i.e. might need your help):

like I said, I am opening two tabs:

• https://vdo.ninja/?view=P1Feed&scene&room=ROOMNAME&password=ROOMPASSWORD in the foreground, and
• https://vdo.ninja/?dir=ROOMNAME&pw=ROOMPASSWORD&showdirector&cleandirector&sticky&push=cdEPXMn in the background
• My hope is that without any user interaction, a few things can happen automatically, namely A) at the moment, the &sticky seems to trigger confusion, and the foreground tab asks whether it should just load the background tabs URL... this might not be avoidable, so do I need to can &sticky? But see my next point, B) I managed to autoplay the foreground tab by adding --autoplay-policy=no-user-gesture-required to the chromium-browser command line, but it seems I cannot force the director tab to automatically publish video, without first manually changing to the tab and clicking on the button enable director's microphone or video - any chance to get a URL switch for that, selecting the default video and audio source and starting the broadcast?

I believe if both A and B are solved, then it is a completely headless setup that allows the giving out of a guest link and is otherwise good to go as soon as it is booted up, no interference necessary!

And, I guess your thoughts on this last one:

Seeing that I am not actually using raspberry_ninja (which seem to be performing slightly worse in fact in terms of audio sync than Chromium), would I be better off to use a different standard package like Jessie or similar, and just add i.e. the ethernet tweak? I mean, I might be wrong, but it seems to me that the modified raspberry_ninja image does not add too much weight, so changing horses might not yield in too much improvement, or am I wrong?

Again, 1000 thanks, and the below just from an historic perspective.

Cheers, Freddy

Hi @steveseguin ,

Thank you for your response, and just clarifying - I am not after multi-casting - each pie would deal with one single outgoing AV stream (as director of the room, using USB HDMI dongle, which receives its input from another computer), and one single incoming stream (the solo link of the remote participant, full screen, and out via its MicroHDMI output as one of several sources to a streaming ATEM switcher) - so we are actually talking 3 rooms (one per pie), with one director and one participant each.

I have since done quite a bit of tinkering, with the following results - I used your image as a starting point and added limited Xorg and gui and chromium, and seem to see fairly OK sync and latency at 720p in both direction - which is an acceptable quality for me. I also experimented with the kiosk image that you recommended, but since we have control over the devices, that was just a little bit too restrictive.

With regards to input devices, I tested with my Elgato Cam Link first, but to scale with that model would be a bit cost prohibitive, and thus tried this one: https://www.amazon.com.au/gp/product/B08CDSXV47

When it runs it works fine, but what I did find a bit annoying (and different to the Elgato) is that it only links with/provides EDID for the source when somebody tries to use it (i.e. a browser), not (like the Elgato) as soon as it has USB power... that has the potential to be a bit disruptive, so just in case you have come across a cheap model that does not have this annoying behaviour, that would be ace to know.

Otherwise, I wonder if it is easy enough to get the director to auto publish video and audio, maybe with a URL switch? At the moment, I have to click myself through to enable it, and my absolute dream would be to automate the entire raspberry to do all of the below in order:

A) load starts B) load chromium in fullscreen (I found the command line switch, but when I try to "startx /usr/bin/chromium-browser ..." with flags, a) there is no Raspbian menu bar available in case needed, and b) vnc gives you an annoying splash screen instead of the subtle menu bar item, so I had hoped I could open chromium automatically AFTER startx, but all attempts at editing ~/.xinitrc seem to fail with an instant crash on start of startx) C) Load tab 1: director link https://vdo.ninja/?dir=ROOMNAME&pw=PASSWORD&showdirector&cleandirector&sticky&push=cdEPXMn (ideally this should automatically connect the cam and the audio as per the last remembered settings, which I hoped &sticky would achieve), and D) Load tab 2: the solo link https://vdo.ninja/?view=P1Feed&scene&room=ROOMNAME&password=PASSWORD forthe single (remote) participant (and, if you are not already, go to full screen) - it seems that the solo link displays a "play" icon in the middle of the page that wants to be clicked first, is that avoidable?

So, long story short (and your patience most appreciated) - any hints how to make A to D work better? And, seeing that I am not actually using raspberry_ninja (which seem to be performing slightly worse in fact in terms of audio sync than Chromium), would I be better off to use a different standard package like Jessie or similar, and just add i.e. the ethernet tweak? I mean, I might be wrong, but it seems to me that the modified raspberry_ninja image does not add too much weight, so changing horses might not yield in too much improvement, or am I wrong?

Certainly, I am feeling like I am getting close to what I need for the setup I aspire, which is great, and again - many thanks for all your hard work on vdo.ninja!

Cheers, Freddy