search

capnmidnight / Calla

Virtual Meetups through Jitsi
https://www.calla.chat
MIT License
1.08k stars 77 forks source link

Potential audio optimizations #43

Closed patcon closed 4 years ago

patcon commented 4 years ago

Love this project!

Seems there may be ways to tweak the noise cancellation features of Jitsi Meet that are causing issue: https://community.jitsi.org/t/higher-audio-quality/31441/6

Have any of these been attempted yet?

capnmidnight commented 4 years ago

Interesting

Here is the code in lib-jitsi-meet where the values are read https://github.com/jitsi/lib-jitsi-meet/blob/dffe94f270f692c7421412bbbc597279e4a2d805/modules/RTC/RTCUtils.js#L77-L90

Gives a little more explanation for what they do.

Seems like you can achieve turning them all off with just setting "disableAP" (disables all audio processing) to true.

I've turned them off, but I don't have a good way to test it and see how it sounds.

capnmidnight commented 4 years ago

User "aseeg" says he was unable to tell a difference after testing, with any setting other than enabling stereo audio. But that also doubles the bandwidth usage. I don't know, this sounds pretty dubious.

patcon commented 4 years ago

Ah, ok! Thanks! Looking through all the sub-settings, really surprised there's no effect. I suppose we could close this?

Hm. Or maybe the final check might be to confirm these findings as matching expectations of Jitsi dev's?

Also, tracing back the constraints object, seems like it might be per-user: https://github.com/jitsi/lib-jitsi-meet/blob/dffe94f270f692c7421412bbbc597279e4a2d805/modules/RTC/RTCUtils.js#L1367

This jibes with another thread I saw, where someone was running a dance party, and only the person playing the music was supposed to toggle those settings...

Think this is worth keeping open and experimenting more, or should we close?

patcon commented 4 years ago

Hey Sean, if I were to describe this to Jitsi devs, with just enough information to help them evaluate/diagnose a potential solution, how would you describe how Calla works? (I understand it to be that sound volume is adjusted at browser level, but the server still sees it as a regular full-volume conference of however many ppl are in the map?)

capnmidnight commented 4 years ago

Yes, the server software is unmodified. There is a small amount of code manually injected into the Jitsi Meet web client that listens for messages from the Calla client (there is no Calla server, all user sync messages are routed through Jitsi Meet). The injected code receives commands to find and reroute the audio element's MediaStream through the WebAudio API to apply a PannerNode, which performs the spatialization.

With spatialized audio, you actually want monaural audio sources, but switching to stereo and down-mixing in the WebAudio API might improve the sampling rate, IDK.

Get Outlook for Androidhttps://aka.ms/ghei36

From: Patrick Connolly notifications@github.com Sent: Friday, May 15, 2020 9:49:42 AM To: capnmidnight/Calla Calla@noreply.github.com Cc: Sean T. McBeth sean.mcbeth@gmail.com; Comment comment@noreply.github.com Subject: Re: [capnmidnight/Calla] Potential audio optimizations (#43)

Hey Sean, if I were to describe this to Jitsi devs, with just enough information to help them evaluate/diagnose a potential solution, how would you describe how Calla works? (I understand it to be that sound volume is adjusted at browser level, but the server still sees it as a regular full-volume conference of however many ppl are in the map?)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/capnmidnight/Calla/issues/43#issuecomment-629246573, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACIYPSHJYTV3CM3ETHJI23RRVB7NANCNFSM4NBACNGA.

capnmidnight commented 4 years ago

I'm currently using what they call the "External API", which runs the standard web client in an IFrame on your page. I'm in the process of converting to either lib-jitsi-meet, or just writing my own WebRTC client code (assuming most of their magic is in the server software, accessed via TURN). Lib-jitsi-meet runs in-page instead of in an IFrame. It will significantly simplify some of the application architecture.

capnmidnight commented 4 years ago

Calla now uses lib-jitsi-meet.

mooomooo commented 4 years ago

Does this mean the jitsi-hax is no longer necessary?

capnmidnight commented 4 years ago

That is correct. Calla now works with the full, default, unmodified install of Jitsi Meet.

capnmidnight commented 4 years ago

Oh, also, the spatialization is now done through Resonance Audio, rather than PannerNode. Quality is significantly better. PannerNode had a lot of crackling it induced in the audio stream.

mooomooo commented 4 years ago

Using lib-jitsi-meet necessitated some changes when using docker; PR #52 addresses those. Ideally nothing should have changed for when not using docker.

capnmidnight commented 4 years ago

using LibJitsiMeet gives us greater control over some features like echo cancellation and such, though it might require writing a hax wrapper around RTCPeerConnection. I could see implementing some UI to let users select whether they're using headphones or their speakers to disable/enable echo cancellation (which is actually mic feedback cancellation, and also just plain doesn't work very well, because I think it's a Google Chrome-only feature).