Initial Support for Spacebar WebRTC

[CE1CECL]

please note: WebRTC was ONLY tested using a test client that was since deleted by spacebar owners, no other clients were tested, nor bot libraries were either (discord.js didn't work for me since it always requires a https connection)

NOTE: This PR DOES NOT add Screen sharing, but adds webcam support and voice BREAKING CHANGES: Windows can no longer be supported after this PR is merged, because medooze-media-server doesn't support windows WebRTC can now be disabled with a env var to continue support for Windows

Known bugs:

• Muting, unmuting, deafening, and undeafening makes the voice call reconnect for unknown reason.
• Defend does not work on test client because of the reconnect bug.
• Video support can be buggy, requiring both cameras on to fully work (why?)
• You tell me what other issues there are: this has been a worked on thing by Me, @MaddyUnderStars & was initially created by @SamuelScheit

[TheKrafter]

*deafen, rather than defend

[Puyodead1]

if there isn't already, there should be a way to disable webrtc from loading, because I use windows for development, and I am not switching anytime soon. either config, or at the very least an environment variable

[TheArcaneBrony]

Worst case you could use WSL I guess? But yeah, this is probably important, even if Windows remains a minimally supported OS.

[CE1CECL]

if there isn't already, there should be a way to disable webrtc from loading, because I use windows for development, and I am not switching anytime soon. either config, or at the very least an environment variable

Added but it needs testing directly on Windows, which I don't have, I have Linux.

[CE1CECL]

Worst case you could use WSL I guess? But yeah, this is probably important, even if Windows remains a minimally supported OS.

No. WSL Is NOT a option, WSL requires Hyper-v which is garbage in general,, slow, etc... WSL is more like emulation than a layer like WineHQ

[TheArcaneBrony]

what? Wine is a stdlib overlay + bincompat layer, WSL is a full on vm (and it isnt slow either)

[CE1CECL]

Marked as ready since its possible to disable webrtc

[MathMan05]

Worst case you could use WSL I guess? But yeah, this is probably important, even if Windows remains a minimally supported OS.

No. WSL Is NOT a option, WSL requires Hyper-v which is garbage in general,, slow, etc... WSL is more like emulation than a layer like WineHQ

WSL 2 needs Hyper-v, the original WSL is to windows what wine is to linux

[TheArcaneBrony]

honestly, hyper-v/WSL2 is completely fine, the slow part is the 9PFS mount for your windows drive

[CE1CECL]

Worst case you could use WSL I guess? But yeah, this is probably important, even if Windows remains a minimally supported OS.

No. WSL Is NOT a option, WSL requires Hyper-v which is garbage in general,, slow, etc... WSL is more like emulation than a layer like WineHQ

WSL 2 needs Hyper-v, the original WSL is to windows what wine is to linux

I wonder if the original WSL will work or not with this? It should, and if it does, then users should use that to enable webrtc until a real native fix comes out.

[MathMan05]

this hasn't had work for quite a while, there's a different WebRTC branch now that has a different blocker, and most of us don't know enough rust to fix it

[CE1CECL]

this hasn't had work for quite a while, there's a different WebRTC branch now that has a different blocker, and most of us don't know enough rust to fix it

Ah, I see! Is there a reason this was re-done from scratch?

[MathMan05]

not 100% sure tbh, I just want to see WebRTC working lol

[CE1CECL]

not 100% sure tbh, I just want to see WebRTC working lol

May I ask what on this PR isnt working exactly?

[MathMan05]

not sure what is wrong with it, I'm not the person who deals with this stuff, I just make a client lol

[dank074]

I just tried this now and it seems to work just fine on WSL at least for testing purposes for those who develop on Windows. Luckily VS Code makes it easy to develop on WSL. I don't know if anyone was ever planning to use Windows for production anyway.

Issues I found:

• Like the PR description states, both cameras have be turned on, but even then, it shows you your own camera instead of the other user's camera. I think it might be doing the same for sound, it's just harder to test sound. EDIT: It's a bug when calling attachTrack you are attaching it to the current client and not the other clients in the call. I changed it and now video is working as expected, even without having both cameras on

• Reconnect bug was solved by puyo in his branch dev/webrtc (his changes on VoiceStateUpdate.ts seemed to solve the issue for this branch as well) changes.patch

• H264 seems to work for me. I just had to modify the SDP.json file and match Discord parameters

• Codec id is hardcoded in SDP.json but it should be dynamic since it will differ between Firefox and Chrome. SelectProtocol message should determine the codec id

• Cannot build Spacebar on Windows since medooze cannot be installed on Windows at all (node-gyp build fails)

I think a possible solution for that last bullet point is probably to make the webrtc server its own separate package which runs independently and communicates with the spacebar instance using RabbitMQ. This will mean that Spacebar will be webrtc-server implementation agnostic, and the end user can choose which implementation to pair with Spacebar. It will also mean that the webrtc server won't necessarily have to be a NodeJS server.

[CE1CECL]

I just tried this now and it seems to work just fine on WSL at least for testing purposes for those who develop on Windows. Luckily VS Code makes it easy to develop on WSL. I don't know if anyone was ever planning to use Windows for production anyway.

Issues I found:

* Like the PR description states, both cameras have be turned on, but even then, it shows you your own camera instead of the other user's camera. I think it might be doing the same for sound, it's just harder to test sound.
  EDIT: It's a bug when calling `attachTrack` you are attaching it to the current client and not the other clients in the call. I changed it and now video is working as expected, even without having both cameras on

May I ask what changes were exactly done?

* Reconnect bug was solved by puyo in his branch dev/webrtc (his changes on VoiceStateUpdate.ts seemed to solve the issue for this branch as well)
  [changes.patch](https://github.com/user-attachments/files/17882959/changes.patch)

I saw this is the same changes as the portion from https://github.com/spacebarchat/server/commit/6beee77a4c14ef7f32486e9c63967efdb2349350 thanks! I'll add it to the branch after I test it and fix the merge conflicts.

* H264 seems to work for me. I just had to modify the SDP.json file and match Discord parameters

* Codec id is hardcoded in SDP.json but it should be dynamic since it will differ between Firefox and Chrome. SelectProtocol message should determine the codec id

* Cannot build Spacebar on Windows since medooze cannot be installed on Windows at all (node-gyp build fails)

I think a possible solution for that last bullet point is probably to make the webrtc server its own separate package which runs independently and communicates with the spacebar instance using RabbitMQ. This will mean that Spacebar will be webrtc-server implementation agnostic, and the end user can choose which implementation to pair with Spacebar. It will also mean that the webrtc server won't necessarily have to be a NodeJS server.

[CE1CECL]

Unless you did this, this is my current guess:

diff --git a/src/webrtc/opcodes/Video.ts b/src/webrtc/opcodes/Video.ts
index aa97c18..c451063 100644
--- a/src/webrtc/opcodes/Video.ts
+++ b/src/webrtc/opcodes/Video.ts
@@ -76,7 +76,7 @@ function createStream(
                if (!client.in.stream) return;

                client.in.stream?.getTracks().forEach((track) => {
-                       attachTrack.call(this, track, client.websocket.user_id);
+                       attachTrack.call(this, track, this.user_id);
                });
        });
 }
@@ -165,7 +165,7 @@ function handleSSRC(this: WebSocket, type: "audio" | "video", ssrcs: SSRCs) {
                        if (client.websocket.user_id === this.user_id) return;
                        if (!client.out.stream) return;

-                       attachTrack.call(this, track, client.websocket.user_id);
+                       attachTrack.call(this, track, this.user_id);
                });
        }
 }

[dank074]

Unless you did this, this is my current guess:

diff --git a/src/webrtc/opcodes/Video.ts b/src/webrtc/opcodes/Video.ts
index aa97c18..c451063 100644
--- a/src/webrtc/opcodes/Video.ts
+++ b/src/webrtc/opcodes/Video.ts
@@ -76,7 +76,7 @@ function createStream(
                if (!client.in.stream) return;

                client.in.stream?.getTracks().forEach((track) => {
-                       attachTrack.call(this, track, client.websocket.user_id);
+                       attachTrack.call(this, track, this.user_id);
                });
        });
 }
@@ -165,7 +165,7 @@ function handleSSRC(this: WebSocket, type: "audio" | "video", ssrcs: SSRCs) {
                        if (client.websocket.user_id === this.user_id) return;
                        if (!client.out.stream) return;

-                       attachTrack.call(this, track, client.websocket.user_id);
+                       attachTrack.call(this, track, this.user_id);
                });
        }
 }

I just did this

diff --git a/src/webrtc/opcodes/Video.ts b/src/webrtc/opcodes/Video.ts
index 1a288f8e..2d646d4b 100644
--- a/src/webrtc/opcodes/Video.ts
+++ b/src/webrtc/opcodes/Video.ts
@@ -165,7 +165,7 @@ function handleSSRC(this: WebSocket, type: "audio" | "video", ssrcs: SSRCs) {
            if (client.websocket.user_id === this.user_id) return;
            if (!client.out.stream) return;

-           attachTrack.call(this, track, client.websocket.user_id);
+           attachTrack.call(client.websocket, track, this.user_id);
        });
    }
 }