polyphony-chat / chorus

A Rust library for interacting with multiple Polyphony- and Spacebar-Compatible instances at once.
https://crates.io/crates/chorus
Mozilla Public License 2.0
16 stars 7 forks source link

Primitive voice implementation (feature/voice) #457

Closed kozabrada123 closed 4 months ago

kozabrada123 commented 8 months ago

(opening this into a pr to track changes, progress, todos and merge conflicts)

This pr (when complete) aims to add voice channel and call support to Chorus via Discord's UDP api.

Current todo:

Features:

Docs:

Compatibility:

Code quality:

UDP vs Webrtc

User accounts sometimes use an undocumented Webrtc api for voice and video, while bots use the documented UDP api. I've decided to implement the UDP api for now, since Webrtc is much more complex and less documented. We may want to look into Webrtc aswell in the future.

Architecture and public API

My current plan is to have four parts of the impl: the Voice Gateway, voice UDP, Voice Data and Voice Handler

The user can then interact with the Voice Gateway to issue gateway commands and receive voice context. The voice UDP handler would expose a slightly higher level api, with listener events for decrypted and deserialized audio as well as a method which serializes, encrypts and sends audio.

kozabrada123 commented 8 months ago

Welp these new build errors on wasm are just because I have no compatibility yet

kozabrada123 commented 8 months ago

Update: I am now be 100% sure I am properly receiving and decrypting the audio

kozabrada123 commented 8 months ago

I've tried to make a very simple app that echoes voice data back, but the sent data never arrives to the original client

kozabrada123 commented 8 months ago

Maybe I messed something up in the sending code or Discord might just have a very sophisticated algorithm :tm: that just ignores packets if they have duplicate data

kozabrada123 commented 8 months ago

Github is drunk, idk why it wanted to merge w/o any new commits on dev

kozabrada123 commented 6 months ago

Update on this, I haven't had the time to work on this for a while, I'll see when I can restart this the next thing I will probably try is encoding and decoding files with a crate

kozabrada123 commented 6 months ago

If I can get decoding and encoding working in an example app, I can see whether the ddc client plays my encoded and serialized audio. if not, then I've messed something up / something is still missing.

If it does play, I can ship the app's code as an example and get to merging this!

kozabrada123 commented 5 months ago

The transport layer works perfectly, I just can't decode ddc's opus for some reason :/