Mixer app : is the most of the load (CPU & Network) always at the mixer (like it was on director) or is it more at the scene view end?

[superseby2]

Hello,

I've discovered the mixer app and played a little bit with it : I realized it might be a good replacement already (eventhough it is quite early stage) to OBS! What I would like to try is to get ridof OBS and use VDOninja from the capture to the streaming end. What I have in mind is that kind of setup :

[ Mixer APP ] <--> [ Browser opening the scene view ] <--> [ ffmpeg grab to RTMP ]

with the part [browser opening the scenview ] + [ FFmpeg ] being headless on a server to take advantage of more CPU/bandwith capabilities.

That's when I realized I am no too sure where the bottleneck is :thinking: : is it where the Mixer App is opened? or where the Scene is rendered?

I've tested this : opening a new room using the Mixer, adding some guests, opening the scene as well.

• I can see everything nicely mixed in the scene browsers.

When I close the Mixer :

• I am still able to see the scene being updated properly.

-> My conclusion : everything is mixed together at the scene end, hence all load must be there?

Thanks so much for the app and for any pointers to make me understand better how all this work.

[steveseguin]

The mixer app itself is pretty light weight, so long as you are not publishing your own video from it to other guests. The Mixer app plays back normally low-quality versions of the video, so it's not heavy and can even be closed. Encoding video is the primary load, as decoding/playback of video is normally handled entirely by the GPU (not CPU).

The scene is also pretty lightweight, but the video that gets played-back is of higher quality, so if your GPU can't keep up, it might stress the CPU out. Most dedicated GPUs should have no problem here. Normally the only CPU load at this point is converting the browser source into a video stream with FFMpeg. If using OBS, you can sometimes get away with NVEnc, putting that encoding load onto the GPU also, rather than the CPU; FFMPEG might also have a hardware-encoder option to reduce CPU load.

On a mac/linux, the efficiency of those systems can sometimes be poor due to poor optimizations in OBS or browser or encoder. I'm mainly familiar with Windows-systems.

I do have some APIs to change layouts of scenes without needing the mixer, so technically the parent of the browser source of the scene you are encoding with FFMPEG can also be used to change the layout. The mixer interface is mainly to make those controls accessible, and to make scenes easier to create/manage.

[superseby2]

Thx steve. And yes I saw the layout commands in the API and was actually thinking on using it as well!

One question though : if the mixer app is not needed in the equation, when every guest is set with "&bc", where are their streams pushed to? The scene I guess?

Extra question ;) : when guests have "&layout" doest it mean they receive the "mixed stream" directly from the Scene? Also you are mentioning that it would be more "Cpu intensive". Is it fair to say that this would be as CPU intensive (and probably less) than when guests are in a "regular meshed-scenario" where each of them are receiving video streams from all participants (while here they send only one stream (theirs) and receive one stream (the mixed ouput))

[steveseguin]

If you are using &broadcast mode (bc) on the guests, they will not see any remote video if the mixer or director is not providing a video back to them. Broadcast mode is configured to only load the mixer/director's video, or some specified video. This does put a lot of load on the director/mixer, as they are encoding a video stream per guest.

To reduce that load, you could share a Meshcast-based view link with the guests in broadcast mode, (or another low latency CDN viewer page). Guests will see a high quality stream, and higher latency, but the director/mixer doesn't have to work so hard. This would only require encoding the video one time for all the guests, as the server would distribute that one stream to all the guests, reducing CPU/Network load on whatever system was generating that feed.

If you don't use a broadcast server (meshcast, etc) or broadcast mode, the guests will run in a mesh-mode, where they can see each other's videos. The director/mixer/server won't be needed in this case. If the guests are using &layout, or are having their layouts remotely controlled, they should see a similar layout as to what the FFMPEG scene gets, but at a lower quality. (as guests get reduced quality per video, in the default mesh mode). Otherwise, if not syncing with the layouts, they will just see the video auto-mixed.

The default mesh-mode does put more stress on each guest, since they need to encode/share more video streams, but in small groups or with guests on powerful computers, it isn't an issue. Guests on older laptops, budget mobile phones, or weak internet connections can sometimes be problematic in mesh-modes.

[superseby2]

OK I see. So the way it works in broadcast mode does not change wether you are using the mixer or the director.

But I guess then that there must be a "fallback" of some sort? Because if I am closing the mixer, the scene remains with the last layout set but video streams are still being refreshed accordingly. :thinking: That's what made me think originally that the load was all on the Scene end. So now I am very much confused. :|

Sorry if those are dumb questions and thank you so much for taking the time to answering them.

[steveseguin]

If the guests are in broadcast mode, and you close the director/mixer, then the guests will lose video.

If the guests are in mesh-mode, the director/mixer can be closed, but you'll need a different way to issue different layouts to the guest if you intend to change the layout. You can do this via the &api, for example, but I'd recommend setting guests to certain slots and selecting layouts as the mixer or director. The mixer/director does not need to share video with the guests when in mesh-mode, so the load on the mixer/director would be near zero if needed.

You can have the guests see a low-latency rendered broadcast from a content delivery network, like meshcast.io, and then display that on a page. VDO.Ninja could be made accessible to the guests via a custom iframe interface, with basic controls to mute and change camera/mic. This is one way to have the mixer/director closed, but still have the guests see the final broadcast without mesh-mode bogging them down.

I've try to support every technical feasible option; I can explore additional ways if needed.

[superseby2]

If the guests are in broadcast mode, and you close the director/mixer, then the guests will lose video.

That's not what I am experiencing. This is how I am testing it using the director but it's the same with mixer:

• Create a room and connect as director [browser 1]

https://vdo.ninja/?director=TESTROOMFORDIRECTOR

• Join as guest 1 from my PC but another browser [browser 2]

https://vdo.ninja/?room=TESTROOMFORDIRECTOR&broadcast

• Launch the scene

https://vdo.ninja/?scene&room=TESTROOMFORDIRECTOR [browser 3]

• Join as guest 2 from my PC but another browser

https://vdo.ninja/?room=TESTROOMFORDIRECTOR&broadcast [browser 4]

=> In [browser 3] I can see the 2 guests

I am closing [browser 1]

• I still can see the 2 guests in browser 3
• I can even add another one still in bc mode and it will show up in browser 3
• The 2 othe browsers 2 & 4 display their local feed.

Below are what is dispayed in my Statistics panel. Worth noting : in both modes (bc & mesh), the number of connections stay the same. (only difference is that in mesh mode the guest do see other guests in his browser).

I am probaly missing something but I don't know what.

network_type
4g / undefined
outbound_connections
3
inbound_connections
2
video_settings
1280x720 @ 30fps
version
21.2
useragent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.3 Chrome/87.0.4280.144 Safari/537.36
platform
Linux x86_64
scene
0
scaleFactor
93%
available_outgoing_bitrate_kbps
4995
average_roundTripTime_ms
1
remote_candidateType
host
local_candidateType
💸 relay server
resolution
1191 x 670 @ 30
video_encoder
libvpx
quality_limitation_reason
none
total_pli_count
0
total_key_frames_encoded
13
audio_bitrate_kbps
1
video_bitrate_kbps
2553
nacks_per_second
0
retransmitted_kbps
0
total_sending_bitrate_kbps
2657
local_relayIP
[X.X.X.X](https://whatismyipaddress.com/ip/X.X.X.X)
local_relayProtocol
udp
average_roundTripTime_ms
105
local_candidateType
💸 relay server
remote_candidateType
💸 relay server
version
21.2
useragent
Mozilla/5.0 (Linux; Android 8.1.0; Moto G (5) Plus) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Mobile Safari/537.36
platform
Linux armv7l
available_outgoing_bitrate_kbps
65
total_sending_bitrate_kbps
23
audio_bitrate_kbps
8
remote_relay_IP
[X.X.X.X](https://whatismyipaddress.com/ip/X.X.X.X)
local_relayIP
[X.X.X.X](https://whatismyipaddress.com/ip/X.X.X.X)
local_relayProtocol
udp
version
21.2
useragent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.3 Chrome/87.0.4280.144 Safari/537.36
platform
Linux x86_64
director
true
scaleFactor
25%
available_outgoing_bitrate_kbps
174
average_roundTripTime_ms
3
local_candidateType
💸 relay server
remote_candidateType
host
quality_limitation_reason
none
total_pli_count
0
total_key_frames_encoded
5
resolution
320 x 180 @ 30
video_encoder
libvpx
video_bitrate_kbps
36
nacks_per_second
0
retransmitted_kbps
0
total_sending_bitrate_kbps
74
audio_bitrate_kbps
13
local_relayIP
[X.XX.X](https://whatismyipaddress.com/ip/X.X.X.X)
local_relayProtocol
udp```

[superseby2]

Update : When I close the browser (and not the tab) of the director, it doesn't show up in other guests'statistic panel. (but the outcome is the same).

[superseby2]

Any chance to tell me what I might do wrong here? Thx!

[steveseguin]

That's not what I am experiencing. This is how I am testing it using the director but it's the same with mixer:

Create a room and connect as director [browser 1]
https://vdo.ninja/?director=TESTROOMFORDIRECTOR

So in this case, the browser 1 is the video source that guests in the room will see, assuming the director has enabled their camera.

Join as guest 1 from my PC but another browser [browser 2]
https://vdo.ninja/?room=TESTROOMFORDIRECTOR&broadcast

This guest should see only the director's video, if available, or just their own local preview video

Launch the scene
https://vdo.ninja/?scene&room=TESTROOMFORDIRECTOR [browser 3]

The browser 3 here is the scene, so it will see everyone in the room since you are using the default scene.

If you use&scene=1 instead, for example, you will only see guests that you added to the scene.

Join as guest 2 from my PC but another browser
https://vdo.ninja/?room=TESTROOMFORDIRECTOR&broadcast [browser 4]

Same as guest 1; they should only see the video of the director, if available.

=> In [browser 3] I can see the 2 guests

I am closing [browser 1]

I still can see the 2 guests in browser 3

browser 3 is the scene link. This will see all the guests in the room. This is expected, as a scene link is not a guest.

I can even add another one still in bc mode and it will show up in browser 3
The 2 othe browsers 2 & 4 display their local feed.

This is expected.

Below are what is dispayed in my Statistics panel. Worth noting : in both modes (bc & mesh), the number of connections stay the same. (only difference is that in mesh mode the guest do see other guests in his browser).

I am probaly missing something but I don't know what.

The scene is requesting videos from the guest, at high quality, to display in OBS or wherever.

The scene will work even if the director room is not open.

You can control the scene via an IFRAME API or even the HTTP/WSS api. You can also configure the scene to show a specific layout or stream IDs via URL parameters.

The scene is just playing back videos, so it is decoding them, but that task is often done by the GPU on the computer. Unless you are on a macbook, which is poorly optimized to use OBS it seems, this shouldn't add much load to the OBS system. Instead, having the scene link will put load on each Guest computer.

If using &broadcast mode on the guest links, the guests will only be sending high quality video to the scene link you have open. It will not send video to other guests.

If the director has their room open, the guests will send a low-quailty stream to the director, so the director can see a preview to help with controlling the stream. The director's tab will only use up significant CPU if they are sharing their video back to the group, as that is the video guests will see if made available. Otherwise, they will just see their own preview.

If the only VDO.Ninja link you have open on the OBS computer is the scene link, and you're seeing high CPU on it, then there might be a problem with your OBS setup or your computer. Video playback should be efficient, but on some mac systems, that isn't always the case. It could also be an issue with the hardware acceleration within your OBS studio not being enabled -- decoding video in software can use up a lot of CPU.

You can use the Electron Capture app, (see my other github projects), as an alternative to OBS browser source for capturing video into OBS. It is often more efficient compared to OBS, and might work better, if OBS isn't working that well.