search

mozilla / janus-plugin-sfu

Janus plugin to act as a kind of SFU for game networking data.
Mozilla Public License 2.0
135 stars 40 forks source link

Publisher in multiple rooms #91

Open vincentfretin opened 2 years ago

vincentfretin commented 2 years ago

I rewrote @devfans changes against master https://github.com/mozilla/janus-plugin-sfu/issues/55#issuecomment-778065876 This includes the commit from #90. I will rebase once #90 is merged. This may not be the final implementation, maybe adding an additional message or modifying an existing message may be better. I post it as draft if anyone want to play with it.

To play with it, in the naf-janus-adapter repo

cp examples/index.html examples/1.html # leave the room as is: room:1
cp examples/index.html examples/2.html # edit 2.html to set room:2
cp examples/index.html examples/12.html # edit 12.html to set room:1-2

open in 3 different tabs:

In the browser console type the following:

First user is in room 1 and room 2:

> NAF.clientId
"3079162068668260"
> NAF.connection.adapter.availableOccupants
[]
> NAF.connection.adapter.mediaStreams
{3079162068668260: {…}}

Second user in room 1:

> NAF.clientId
"2058753888545049"
> NAF.connection.adapter.availableOccupants
["3079162068668260"]
> NAF.connection.adapter.mediaStreams
{2058753888545049: {…}, 3079162068668260: {…}}

Third user in room 2:

> NAF.clientId
"2094703467261257"
> NAF.connection.adapter.availableOccupants
["3079162068668260"]
> NAF.connection.adapter.mediaStreams
{2094703467261257: {…}, 3079162068668260: {…}}

So all availableOccupants and mediaStreams are what we expect. Avatars and properly hearing the audio of those avatars is another thing.

If second user (room 1 tab) connected after the first user (publisher in tab 12.html), issue this command for the first user

document.getElementById('player').components.networked.syncAll(null, true)

to force sending the entities (the avatar) to participants of main room (room 1), this will create the publisher avatar (and so the networked-audio-source) in room 1 tab.

Why the publisher avatar appears in room 1 tab when it connects after the others? Because the entities are sent with this.syncAll(undefined, true); on each networked component via the connected listener when the user connect. The other case where the entities are sent is when a datachannel open when we subscribe to a user, it calls this.entities.completeSync(clientId, true) that calls syncAll on each entity, this is not the case here because the publisher subscribe to no one, this is why the publisher avatar doesn't appear if a participant goes into room 1 after the publisher.

For room 2: We never see the first user (publisher) avatar (so no audio) because the naf updates doesn't go to room 2 (because we use datachannel transport and we have &joined.room_ids[0] in data_recipients_for used by incoming_data). The first user (publisher) mediastream is there though, you can listen to it:

let publisherId = Object.keys(NAF.connection.adapter.mediaStreams).filter(key => key != NAF.clientId)[0]; // take the first mediastream that is not me, this works only with two users in the room
const audio = document.createElement('audio');
audio.srcObject = NAF.connection.adapter.mediaStreams[publisherId].audio;
audio.play();

In 12.html tab, there are some strange things happening but explainable:

Now if we use websocket transport instead of datachannel. In the html files, add in adapter-ready listener:

adapter.unreliableTransport = "websocket";
adapter.reliableTransport = "websocket";

instead of incoming_data, this will use process_data that broadcasts naf updates to all rooms the publisher is in, so creating the publisher avatar in room 2 tab as well (the issue of syncAll explained above still applies). But we still have the issue of ghost avatars in publisher room (12.html). This needs changes in naf-janus-adapter where we have the various this.room checks. The publisher doesn't hear others, that's expected for performance but you may want to at least hear the users in the main room (room 1).

So yeah here is the state of things. You will need changes in naf-janus-adapter and in your app to make use of this for a given use case. The function incoming_data should probably do the same thing that process_data to broadcasts to all rooms I think, it doesn't make sense that it's different, well it may depend of the use case. I don't have a clear use case myself for now. :)

vincentfretin commented 2 years ago

See https://github.com/networked-aframe/naf-janus-adapter/pull/15 if you want to play with the example I described.

vincentfretin commented 2 years ago

The main issue I see is that the publisher (speaker) is receiving naf updates (u, um, r messages) from all avatars from all rooms, this may be a big perf issue for the publisher if they publishes in 10 rooms with 20 avatars each. You can ignore on the client side the messages by setting NAF.options.syncSource="ignore-all" (this is really a hack so it triggers this code path) but you're still receiving naf updates you don't need. The publisher won't see any avatars and so won't hear any participants in this case.

So the changes here may be interesting only with an alternative transport other than "datachannel" (which uses incoming_data) or "websocket" (which uses process_data) transports for the naf updates, like Phoenix channel that Hubs is using (here in hubs and here in reticulum) so the naf messages only broadcast to the room channel you're in (room "1" for example), but you can still broadcast the audio in other rooms by setting networked-scene room (room "1-2-3-4-5" to broadcast to rooms 1,2,3,4,5 for example)

vincentfretin commented 2 years ago

While I'm thinking about it, I'm adding a note here. If you have some horizontal scaling implemented in your infra to scale the rooms on several janus instances, you need to make sure all the rooms 1,2,3,4,5 goes into the same janus instance.

vincentfretin commented 2 years ago

After some thinking, some changes I may do to this PR: