Can't Achieve High-Fidelity Audio for Music Streaming

[ForemanDev]

Currently it is not possible to achieve high-fidelity audio quality for music streaming using the LiveKit Unity Web SDK. I discussed this in Slack with @theomonnom, @noahlt, and @davidzhao and am filing an issue report after confirming the bug.

This issue is quite problematic for our team because when someone tries to stream music via screen share (for example a DJ streaming a live performance into a virtual world) the music quality is very "garbled-sounding" and low quality and there is currently no way to fix it.

I am aware of the Hi-fi audio documentation, however this documentation does not cover how to implement hi-fi audio in the Unity Web SDK. I attempted to translate the documentation into Unity C# as best I could, however when doing so this causes screen share to break (no errors or warnings, screen share simply does not initialize at all).

Here is my working code to start a screen share with audio (not hi-fi audio):

private IEnumerator BeginScreenShareRoutine()
{
    TrackPublishOptions trackPublishOptions = new TrackPublishOptions
    {
        Name = "ScreenShare_" + GameManager.Instance.GetUserID()  // Append the instigator's user ID to the name of the screen share.
    };

    ScreenShareCaptureOptions screenShareCaptureOptions = new ScreenShareCaptureOptions
    {
        Audio = true    // *Need* to set 'Audio = true' in order for screen share to include audio.
    };

    yield return room.LocalParticipant.SetScreenShareEnabled(true, screenShareCaptureOptions, trackPublishOptions);
}

The above code works fine to initiate a screen share - the window pops up and you can choose what to share and whether or not to include system audio. Below is my attempt to adjust the settings to use high-fidelity audio:

private IEnumerator BeginScreenShareRoutineNew()
{
    TrackPublishOptions trackPublishOptions = new TrackPublishOptions
    {
        Name = "ScreenShare_" + GameManager.Instance.GetUserID(),  // Append the instigator's user ID to the name of the screen share.
        Source = TrackSource.ScreenShare,

        videoEncoding = new VideoEncoding{ MaxBitrate = 1_700_000, MaxFramerate = 30 },
        ScreenShareEncoding = new VideoEncoding { MaxBitrate = 2_500_000, MaxFramerate = 15 },
        VideoCodec = LiveKit.VideoCodec.VP8,
        AudioBitrate = 32_000,
        DTX = false,
        Simulcast = true,
        StopMicTrackOnMute = false
        //VideoSimulcastLayers = ,  // ???
        //ScreenShareSimulcastLayers =  // ???
    };

    ScreenShareCaptureOptions screenShareCaptureOptions = new ScreenShareCaptureOptions
    {
        Audio = true,    // *Need* to set 'Audio = true' in order for screen share to include audio.
        AudioOptions = new AudioCaptureOptions    // *** Setting an AudioCaptureOptions within ScreenShareCaptureOptions BREAKS screen share. ***
        {
            AutoGainControl = true,
            ChannelCount = 2,
            //DeviceId = , // ???
            EchoCancellation = false,
            //Latency = ,   // ???
            NoiseSuppression = false
            //SampleRate = ,    // ???
            //SampleSize =      // ???
        }
        //Resolution = new VideoResolution
        //{
            //Width = , // Does all of this get automatically set?
            //Height = ,
            //FrameRate = ,
            //AspectRatio = 
        //}
    };

    yield return room.LocalParticipant.SetScreenShareEnabled(true, screenShareCaptureOptions, trackPublishOptions);
}

The above code breaks screen share. There is no error or warning - screen share simply does not initialize at all. No window pops up when I start the above coroutine.

Hopefully this is a fairly simple issue to fix? I assume some wires just need to be connected up from C# to JS in order to bring things in line with the Hi-fi audio documentation? And hopefully this would be a good opportunity to update the documentation to include Unity. :)