sjoerdvankreel
commented
8 months ago Hey!
You should be able to use an aggregate stream for this. Unfortunately I cannot test this ATM because my current setup doesn't have the required audio inputs. Best I can do is try to aggregate headset input with a wasapi loopback stream on the speakers, but unfortunately there is no single sample rate that is supported by both devices, and neither xtaudio or wasapi do sample rate conversion so i'm out of luck. The C# example below should get you started, but since i cannot test it, it's probably full of bugs. On a side note, you should never do I/O in the audio callback. No file I/O, no websockets etc etc.
using System;
using System.IO;
using System.Threading;
using Xt;
namespace ConsoleApp18
{
internal class Program
{
[STAThread]
static void Main(string[] args)
{
using var audio = XtAudio.Init(nameof(Program), 0);
var service = audio.GetService(XtSystem.WASAPI);
using var list = service.OpenDeviceList(XtEnumFlags.Input);
for (int i = 0; i < list.GetCount(); i++)
Console.WriteLine(list.GetName(list.GetId(i)));
// These are the indices for my system:
int microphoneInIndex = 0; // Headset Microphone (Jabra EVOLVE 20 MS) (Shared)
int speakerLoopbackIndex = 5; // Speakers / Headphones (Realtek Audio) (Loopback)
using var microphoneInDevice = service.OpenDevice(list.GetId(microphoneInIndex));
using var speakerLoopbackDevice = service.OpenDevice(list.GetId(speakerLoopbackIndex));
var mix = new XtMix();
// my mic only supports 16 khz
// but the loopback device does NOT support 16khz so i cannot test
mix.rate = 48000;
mix.sample = XtSample.Float32;
var deviceParams = new XtAggregateDeviceParams[2];
deviceParams[0] = new XtAggregateDeviceParams();
deviceParams[0].bufferSize = 20;
deviceParams[0].channels.inputs = 2;
deviceParams[0].device = microphoneInDevice;
deviceParams[1] = new XtAggregateDeviceParams();
deviceParams[1].bufferSize = 20;
deviceParams[1].channels.inputs = 2;
deviceParams[1].device = speakerLoopbackDevice;
var streamParams = new XtStreamParams(false, OnBuffer, null, null);
var aggregateParams = new XtAggregateStreamParams(in streamParams, deviceParams, 2, in mix, microphoneInDevice);
using var stream = service.AggregateStream(in aggregateParams, null);
using var safe = XtSafeBuffer.Register(stream);
_mixdownBuffer = new float[2 * stream.GetFrames()];
stream.Start();
for(int i = 0; i < 10; i++)
{
Console.WriteLine(i + 1);
Thread.Sleep(1000);
}
stream.Stop();
var byteBuffer = new byte[2 * 48000 * 10 * sizeof(float)];
// 2 = stereo
Buffer.BlockCopy(_mixdownBuffer, 0, byteBuffer, 0, _mixdownPosition * 2 * sizeof(float));
File.WriteAllBytes("C:\\temp\\mixdown.raw", byteBuffer);
}
static int _mixdownPosition = 0;
static float[] _mixdownBuffer = new float[2 * 48000 * 10];
static int OnBuffer(XtStream stream, in XtBuffer buffer, object user)
{
var safe = XtSafeBuffer.Get(stream);
safe.Lock(in buffer);
var input = (float[][])safe.GetInput();
// _mixdownBuffer.Length / 2 because stereo
for (int f = 0; f < buffer.frames && _mixdownPosition < _mixdownBuffer.Length / 2; f++)
{
// non-interleaved
float microphoneLeft = input[0][f];
float microphoneRight = input[1][f];
float speakerLoopbackLeft = input[2][f];
float speakerLoopbackRight = input[3][f];
// interleaved
_mixdownBuffer[_mixdownPosition * 2 + 0] = (microphoneLeft + speakerLoopbackLeft) / 2;
_mixdownBuffer[_mixdownPosition * 2 + 1] = (microphoneRight + speakerLoopbackRight) / 2;
_mixdownPosition++;
}
safe.Unlock(in buffer);
return 0;
}
}
}
liuuuu-pro
Hello, this is a powerful library, but I am currently facing a problem. I want to record both microphone audio and speaker audio at the same time, with the speaker audio being recorded through LOOPBACK mode. How can I more scientifically mix the microphone audio and the speaker audio together? Can I use XtServiceAggregateStream?
I am currently recording audio from two separate devices and then using ffmpeg for mixing. However, I want to obtain a real-time mixed audio stream. How can I achieve this?
I have looked at the Aggregate example, but it aggregates the microphone's input audio to the speaker's output audio. However, this is not what I need. I need to record the audio from both the microphone and the speaker.