search

martindevans / MumbleSharp

An implementation of the mumble voice chat protocol in C#
http://martindevans.me
MIT License
91 stars 28 forks source link

Optional Pure C# Implementation of Opus Codec #51

Open trevorirwin opened 4 years ago

trevorirwin commented 4 years ago

Per the Readme, I wanted to create an issue to share what I'm working on.

My end goal is to use MumbleSharp in a Xamarin application for iOS and Android. However, the current framework MumbleSharp has set up for loading binaries does not appear to be compatible with iOS, to the best of my understanding. I am admittedly not very experienced with native code in C#, and even less so with iOS, so it may yet be possible to adapt the current system to mobile.

edit: It's absolutely possible to reference static libraries on iOS, but I think it will require some refactoring of NativeMethods.cs, or a mobile-specific implementation thereof, anyway.

However, rather than pursue that route, I decided to seek out a C# implementation of the Opus Codec. Concentus is one implementation I found. They claim 40-50% of native performance, which I believe on modern devices (at least for my end use case) should be acceptable. The license looks compatible with MumbleSharp's as well.

My chosen pathway for integration into MumbleSharp was to set it up as a new "codec", in parallel to the existing codecs. Then, in CodecSet.cs, I check the current platform and return the Concentus "Codec" instead of Opus.

The bulk of the integration is already done. I've tested with the MumbleGuiClient built for Windows, forcing GetCodec to always return the Concentus "codec", and so far it works great! Well, I know it works great for encoding; my transmissions to other clients sound crystal clear.

However, even with the regular Opus codec, when I build MumbleGuiClient for Windows, I get extremely choppy audio. That is, I hear a few ms at a time broken up by silences. I think this is related to #46, but I'm not experienced enough to say. Can anyone confirm that my symptoms are indeed "jitter"? If so, I'll also start to tackle the jitter buffer problem. When that's done, I can get back to testing decoding and playback with Concentus.

My ongoing work is available here.

edit: Upon further investigation, I really don't think it would be that hard to use the libopus binaries on iOS (see above), but it might be nice to have the pure C# implementation of Opus available as an option anyway, so I still intend to continue work on this.

martindevans commented 4 years ago

Perhaps an option could be to add the appropriate build switches into nativeMethods.cs so that it switches to using the correct method for a platform and if no natively supported platform is identified it can fall back to using pure C# Opus.

trevorirwin commented 4 years ago

FYI, with my work on the jitter buffer I was able to confirm that this C# implementation works for both playback and recording (including on iOS!) As you said, we'll want a more refined system for detecting whether or not to use it before submitting a PR. I'll hopefully get around to that cleanup work soon as part of my app development.