Don't subscribe to audio/video if occupant is a listener

[vincentfretin]

Don't subscribe to audio/video if occupantId ends with '-l' meaning they are only listener and doesn't talk. The avatar position and chat are still working because datachannel is send/receive on the publisher RTCPeerConnection connection.

Less sessions (RTCPeerConnection) are created on the server, so normally less cpu. That's a hacky and simple way of doing some of the use case I talked about in https://github.com/networked-aframe/janus-plugin-sfu/pull/6 We still use a RTCPeerConnection for the listeners, if we could just use websocket and no RTCPeerConnection at all that would be even better, that would need changes in the rust code for sure, but I don't think that's possible to use janus websocket without having a session.

cc @arpu

[vincentfretin]

If a listener leaves, you get the following errors

networked-aframe.min.js:1 Error getting media stream for 3818317759-l The user disconnected before the media stream was resolved.

that would be spamming sentry. So probably make a change in networked-audio-source to not call NAF.connection.adapter.getMediaStream if ownerId ends with "-l".

[vincentfretin]

We still use a RTCPeerConnection for the listeners, if we could just use websocket and no RTCPeerConnection at all that would be even better, that would need changes in the rust code for sure, but I don't think that's possible to use janus websocket without having a session.

Looking at the code again, I may have had some misunderstanding. In js code, we have the ws, a session object that wraps it and mj.JanusPluginHandle(this.session) to create a handle, then a RTCPeerConnection is associated to the handle. handle.attach is actually using the ws to send the message. On rust side, we get a (handle: *mut PluginSession) (PluginSession is the janus_plugin_session C struct) in the various functions of janus plugin interface that is equivalent of the js JanusPluginHandle, and rust Session is a container for this C handle, so that's completely different meaning than the js this.session. We don't really have access directly to the RTCPeerConnection object in rust, all is done through the plugin api and janus_plugin_session. So it may seems we can just use websocket to be in a room, creating a JanusPluginHandle but not associating a RTCPeerConnection.

I asked chatgpt: With janus-gateway can we create a plugin that use a janus_plugin_session without a webrtc connection, just using websocket?

Yes, with Janus Gateway you can indeed create a plugin that uses a janus_plugin_session without setting up a WebRTC connection, and solely communicate using WebSockets. This approach might be useful for scenarios where the real-time media capabilities of WebRTC aren't required, but you still want to leverage the infrastructure and plugin architecture of Janus for message passing or other interaction patterns.

Here’s how you can approach this:

1. WebSocket Connection Establishment: Use WebSockets to establish a connection between the client and the Janus server. This connection will handle signaling without the media transport capabilities that WebRTC offers.

2. Janus Plugin Development: Develop your custom plugin in Janus. This plugin will not need to handle any WebRTC-related tasks (like managing ICE candidates, SDP offers/answers, etc.). Instead, it will focus on processing the messages received via WebSockets.

3. Session Handling: The plugin can still create and manage janus_plugin_session instances. These sessions will manage the state and context for each connected client. Even without WebRTC, these sessions are useful for keeping track of different users or interactions.

4. Message Handling: Implement the logic for handling messages received from the WebSocket connection within your plugin. Since you are not using WebRTC, all interactions with the plugin will be based on custom or predefined signaling messages you define.

5. Client-Side Implementation: On the client side, you would interact with Janus using WebSocket messages. Your client application would need to handle JSON messaging to start, maintain, and terminate sessions with the Janus server.

6. Deployment and Scaling: Since this setup doesn’t use WebRTC, it might be lighter on bandwidth and processing requirements, but you'll still need to handle WebSocket connections efficiently, especially if scaling to many users.

This architecture leverages the flexibility of Janus and allows you to use its plugin and session management capabilities in applications where real-time media is not necessary. This could be useful in various scenarios, such as command and control platforms, gaming backends, or other interactive services where media streams aren’t required.

[vincentfretin]

Instead of the "-l" suffix hack, we could add a default option to not subscribe to media by default with NAF.options.subscribeToMediaByDefault=false, and modify the addOccupant method to be addOccupant(occupantId, subscribeToMedia=NAF.options.subscribeToMediaByDefault) this way if you do an audio zone, you call addOccupant(clientId, true) with networkedAudioSources[clientId].createSound() when you enter and clientId is in the audio zone with you and addOccupant(clientId, false) with networkedAudioSources[clientId].destroySound() when you leaves the audio zone. The addOccupant function actually call removeOccupant before adding again so that's fine calling it again. Instead of maintaining a networkedAudioSources array of some sort, probably better to emit a ('subscribed-to-audio', {clientId}) and ('unsubscribed-to-audio', {clientId}) events in addOccupant and listen to those in networked-audio-source.

See my comment about createSound()/destroySound() in https://github.com/networked-aframe/networked-aframe/pull/330#issuecomment-1080444549

[vincentfretin]

If a listener leaves, you get the following errors

networked-aframe.min.js:1 Error getting media stream for 3818317759-l The user disconnected before the media stream was resolved.

that would be spamming sentry. So probably make a change in networked-audio-source to not call NAF.connection.adapter.getMediaStream if ownerId ends with "-l".

I actually made an option networked-audio-source="createSoundOnInit:false" to not call createSound() that calls getMediaStream for use with an audio zone.

[arpu]

@vincentfretin we do not need the replace for loop

parseInt("123-l")
123

[vincentfretin]

I replace it for the createPublisher, but it shouldn't be needed for createSubscriber because we don't call it if the clientId ends with "-l". I may have missed something though, I didn't test with an loops config.

[vincentfretin]

oh sorry I misread, we don't need it, ok, I'll remove it.