polyphony-chat / chorus

A Rust library for interacting with multiple Polyphony- and Spacebar-Compatible instances at once.
https://crates.io/crates/chorus
Mozilla Public License 2.0
17 stars 7 forks source link

Voice Channel support #49

Open bitfl0wer opened 1 year ago

bitfl0wer commented 1 year ago

Add support for sending and receiving live streams of WebRTC Video and Audio.

kozabrada123 commented 1 year ago

At least this is well documented

Some relevant info:

(for encryption, see this part of discord's docs)

The encryption and encoding seems to be the hardest part of implementing this, the actual communication is very similar to the gateway

kozabrada123 commented 1 year ago

Oh and we also have no info about video, so we should probably implement voice first

kozabrada123 commented 1 year ago

Doing some research, video uses VP8 and VP9, maybe env-libvpx-sys in rust?

{
    "codecs": [
        {
            "name": "opus",
            "type": "audio",
            "priority": 1000,
            "payload_type": 109,
            "rtx_payload_type": null
        },
        {
            "name": "VP8",
            "type": "video",
            "priority": 2000,
            "payload_type": 120,
            "rtx_payload_type": 124
        },
        {
            "name": "VP9",
            "type": "video",
            "priority": 3000,
            "payload_type": 121,
            "rtx_payload_type": 125
        }
    ]
}
kozabrada123 commented 1 year ago

Webrtc-rs may also be worth looking into, since it seems to support all the needed codecs

kozabrada123 commented 1 year ago

Potentially maybe perhaps looking into this on feature/webrtc I probably won't manage to implement it though lol

bitfl0wer commented 1 year ago

Good luck :)

kozabrada123 commented 1 year ago

Also looking at how Serenity implemented this, they seem to have developed their own rtp parser and even their own opus bindings

bitfl0wer commented 1 year ago

I wonder if we could... yoink this :D

kozabrada123 commented 1 year ago

yup :D

kozabrada123 commented 1 year ago

Slight change of plans, raw UDP first then webrtc because webrtc is really complex and raw udp seems to be way easier to implement for now

kozabrada123 commented 10 months ago

wasm cannot support udp.

kozabrada123 commented 10 months ago

See https://github.com/webrtc-rs/webrtc and https://github.com/wasm-peers/wasm-peers

kozabrada123 commented 1 month ago

A few notes: