How to Add Custom Audio Buffer to the Recorded Audio in Unity

[BasilDavid]

Hi everyone,

I'm currently working on a project where I need to add a captured audio buffer from Agora to the recorded audio in Unity. VideoKit allows recording from Unity's AudioSource or AudioListener, but my specific use case is challenging because Agora is already rendering the audio natively outside of Unity, so I can't pass it through Unity's audio system.

My current workaround involves modifying VideoKit directly in the AudioComponentSource.cs file. specifically the inner class AudioSourceAttachment:

private sealed class AudioSourceAttachment : MonoBehaviour {

    public Action<float[], int>? sampleBufferDelegate;

    private void OnAudioFilterRead(float[] data, int channels) {
        sampleBufferDelegate?.Invoke(data, channels);
    }
}

The idea is to merge my Agora audio buffer with the ongoing Unity audio buffer inside the OnAudioFilterRead method.

However, I'm encountering various issues while editing the library, and I'm wondering if there's an easier or more effective way to achieve this without modifying VideoKit extensively.

Has anyone dealt with a similar situation or have any suggestions on how I might approach this?

Thanks in advance for your help!

[olokobayusuf]

@BasilDavid are you using the VideoKitRecorder component to record? If you aren't (i.e. using the MediaRecorder class directly), then you can manually mix audio from Agora and Unity, then commit the result with MediaRecorder.Append(AudioBuffer). You shouldn't need to modify VideoKit at all.

We have to extend the VideoKitRecorder component to support users providing their own custom recording inputs to use for recording.

[BasilDavid]

@olokobayusuf Thank you for your response! I tried using MediaRecorder.Append(AudioBuffer), but it caused the audio track to become twice the length of the video. This happens because I'm appending my custom buffer, while Unity's AudioListener is already included. When I omit the AudioListener, my audio plays clearly, which suggests that Append is adding to the audio track rather than merging overlapping packets.

I'm using the same clock to generate timestamps, but I'm wondering if there's a way for MediaRecorder.Append(AudioBuffer) to handle and merge overlapping packets instead of just appending them.

Any advice on this would be greatly appreciated!

[olokobayusuf]

The MediaRecorder.Append(AudioBuffer) function (and the overload for appending pixel buffers) will append in time. The recorder doesn't do any form of mixing or merging; you have to implement this yourself and commit the mixed audio to the recorder.

[BasilDavid]

Thank you so much for the deep clarification , I will implement the merging my self before I append the buffer.