Multi-party support, detecting client dropoff?

[thor-schueler]

@steveseguin : first of all, thanks for the awesome work with vdo.ninja. I have been building off your work to create a fully headless IoT Camera and added some additional configuration, multi-party support and management via Azure IoT. I have also dockerized the whole thing.

I have not created a pull because multi-party support required quite a bit of refactoring, so if you want to, please take a look at https://github.com/Avanade/2DOF-PI-WebRTC-Camera. Let me know if you want me to do a pull request. The integration into Azure IoT is optional and can easily be removed for a pull request.

Anyway... one problem that remains and I have not been able to figure that one out. I am not a WebRTC or gstreamer expert by any stretch, so I thought I ask you: Receiving an incoming request, adding a branch to the gst pipeline for the peer and negotiating the connection is no problem. But... when the peer drops off (user closes the browser) there is no signal to the serving peer. As a result, I cannot remove the pipeline branch associated with the peer, which quickly leads to resource exhaustion due to dead branches (since we are still happily pushing packets and using sockets). I looked at the webrtcbin get-stats signal, but the stats do not seem to reflect abandoned peers. The stats show continued packet delivery even though no one is home on the other side.

I was wondering whether you have any ideas on how abandoned peers could be detected. If we could figure that out, we might have a pretty scalable and stable solution.

I was also wondering about using backup.obs.ninja. While that works well, it would be great if we could use the mainline as signaling server.

As an aside, the exactly same approach will work for a Jetson of course.

[steveseguin]

Hi Thor,

I put up on a branch my Gstreamer code for connection detection; it only currently works with the vdo.ninja/beta release of the site, v19, which I'm pretty much ready to release into main this week or so.

Link to the branch/file is here: https://github.com/steveseguin/raspberry_ninja/blob/advanced_example/nvidia_jetson/server.py (I used gst-1.18.4)

The code uses webRTC data-channels to ping/pong the remote peer to ensure the remote connection is detected. Gstreamer/Libnice do not properly support connection detection, but the code for that is included as well -- it just doesn't work because of Gstreamer/Libnice are still very much lacking.

Yep, you can use the production handshake server; wss://wss13.obs.ninja:443. It's intended only for webrtc handshaking though; once a peer connection is established, data-channels are the primary coms thereafter. So long as it doesn't get flooded or abused, feel free to use it.

Thank you for opening your project code with the community; while I probably won't do a pull-request of it, I might dig thru the code later to see if I can see anything of interest to add to this project. Hope it's been fun so far; I'm super excited to see where your project leads!!

[thor-schueler]

Thank you so much Steve! I will check it out. As you go through my project, particularily look at https://github.com/Avanade/2DOF-PI-WebRTC-Camera/blob/master/modules/obscam/app/server.py, methods add_client() and __remove_client(). They take care of managing the gst pipeline based on incoming requests. They should pretty much plug into your code. Also take a look at start_pipeline and how is use it. Essentially, all the pipelines end with a tee and a fake sink. That allows me to start the pipeline as soon as the loop enters. When new connections come, I then call __add_client() to branch off the tee and connect a new webrtcbin...

[thor-schueler]

Thanks for those suggestions! I have implemented the disconnect protocol. It gave me some additional ideas for fallbacks. At the moment, looks like at the moment wss://wss13.obs.ninja:443 does not yet have the ping. Looking forward to testing some more when it does :)

[steveseguin]

For clarification, the websocket server does not ping; please do not ping it as its purely intended for the handshake. The code added to the script pings the remote connection via data channels over the webrtc connection. This ping back is supported on vdo.ninja/beta as a client, which is what pongs back.

[thor-schueler]

ahh.. yes, I got that about not pinging the web socket server. So no worries there. Sorry for the confusion I created. I didn't realize that vdo.ninja and vdo.ninja/beta re using the same socket server.

[steveseguin]

all good. :) Hope things are you going well with development; looks like you've been busy.

[thor-schueler]

Yes :) I have done some testing against vdo.ninja/beta re the heartbeat. I think there might be an issue on the vdo.ninja side though. I think I am wiring up the data channel for the appropriate webrtcbin correctly. After requesting the channel, I am getting a successful response on on_data_channel_open. However, when I send the ping, I never get a response message from the peer browser. However, I am seeing the following javascript error in the peer popping up when I send the message on the data channel:

Uncaught ReferenceError: mustUpdateuserList is not defined at RTCDataChannel._0x31eee2.rpcs..receiveChannel. (webrtc.js?ver=220:1) _0x31eee2.rpcs..receiveChannel. @ webrtc.js?ver=220:1

Is this indeed an issue on the vdo.ninja side or am I doing something wrong? I pretty much applied the code from the advanced example, just extended it to work with the multiple client deal. I looked at webrtc.js in vdo.ninja/beta for the ping/pong protocol implementation, but was not able to find it. Any advice or pointers?

[steveseguin]

Can you do a hard browser refresh and try again? I added some better error handling to beta.

I don't really quite know why your browser threw an error and my chrome tolerated it, but I think I fixed that particular issue.

Your gstreamer should look like this:

And for reference, this is the code for the ping/pong. I also turned on warnlog() output on beta for now.

if (typeof(e.data) == "object"){ 
    // do some binary stuff
} else {
    var msg = JSON.parse(e.data);
    var mustUpdateMixer = false;
    var mustUpdateUserList = false;

    if ("bye" in msg){
        warnlog("BYE");
        session.closeRPC(UUID); // user is telling us they are quitting, so lets clean up preemptively.
        return;
    } else if ("ping" in msg){
        var data = {};
        data.pong = msg.ping;
        session.sendRequest(data,UUID);
        warnlog("PINGED")
        return;
    } else if ("pong" in msg){
        warnlog("PONGED")
        return;
    } 
}

[thor-schueler]

@steveseguin : Thanks so much Steve, it works quite beautifully now :) Heartbeat handshake works like a charm and the pipeline management on the streamer side detects the abandoned clients and removes them from the GST pipeline. This works really lovely now.

I originally used the LattePanda 5MP USB Autofocus Camera with tilt and pan via independent servos, but now I got this Arducam PTZ camera, which is really lovely as it support motorized focus and zoom as well as Pan and Tilt. Adding support for that one into my controller workload next.

I'll publish all the 3-D parts later today I think. Thanks for all the help on getting this to work so well with ninja! I'll close this issue :)