C# language support

[GDjkhp]

i wanna try to implement xt-audio on unity3d, is it possible?

[sjoerdvankreel]

Not sure what you mean? You can probably use them side by side though.

[GDjkhp]

i mean i wanna use it in c# programming language, just like java

[sjoerdvankreel]

Sure that's possible, xtaudio has .net bindings. Supports both .net framework and core. See nuget package here: https://www.nuget.org/packages/Xt.Audio/. Just keep in mind supported backends are x86/64 windows and linux only, whereas unity looks like it supports a lot more platforms.

[GDjkhp]

it gives me an error, dll not found

I installed it on NuGetForUnity plugin

and then I ported your code

`
using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour
{
    // intermediate buffer
    public byte[] BUFFER = new byte[2048];

    // audio streaming callback
    public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
    {
        XtSafeBuffer safe = XtSafeBuffer.Get(stream);
        // lock buffer from native into java
        safe.Lock(buffer) ;
        // short[] because we specified INT16 below
        // this is the captured audio data
        short[] audio = (short[])safe.GetInput();
        // you want a spectrum analyzer, i dump to a file
        // but actually never dump to a file in any serious app
        // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
        processAudio(audio, buffer.frames);
        // unlock buffer from java into native
        safe.Unlock(buffer);
        return 0;
    }

    void processAudio(short[] audio, int frames)
    {
        // convert from short[] to byte[]
        for (int frame = 0; frame < frames; frame++)
        {
            // for 2 channels
            for (int channel = 0; channel < 2; channel++)
            {
                // 2 = channels again
                int sampleIndex = frame * 2 + channel;
                // 2 = 2 bytes for each short
                int byteIndex0 = sampleIndex * 2;
                int byteIndex1 = sampleIndex * 2 + 1;
                // probably some library method for this, somewhere
                // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
                BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
                BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

                /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
                pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                        / 32767.0f;*/
            }
        }

        // by now BYTES contains the data you want,
        // but be sure to account for frame count
        // (i.e. not all off BYTES may contain useful data,
        // might be some unused garbage at the end)

        // compute total bytes this round
        // = frame count * 2 channels * 2 bytes per short (INT16)
        int byteCount = frames * 2 * 2;
    }

    // Start is called before the first frame update
    void Start()
    {
        // this initializes platform dependent stuff like COM
        XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);
        // works on windows only, obviously
        XtService service = platform.GetService(XtSystem.WASAPI);
        // list input devices (this includes loopback)
        XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input);
        for (int i = 0; i < list.GetCount(); i++)
        {
            String deviceId = list.GetId(i);
            XtDeviceCaps caps = list.GetCapabilities(deviceId);
            // filter loopback devices
            if (caps.Equals(XtDeviceCaps.Loopback))
            {
                String deviceName = list.GetName(deviceId);
                // open device
                XtDevice device = service.OpenDevice(deviceId);
                // 16 bit 48khz
                XtMix mix = new XtMix(48000, XtSample.Int16);
                // Structs.XtMix mix = new Structs.XtMix(44100, Enums.XtSample.INT16);
                // 2 channels input, no masking
                XtChannels channels = new XtChannels(2, 0, 0, 0);
                // final audio format
                XtFormat format = new XtFormat(mix, channels);
                // query min/max/default buffer sizes
                XtBufferSize bufferSize = device.GetBufferSize(format);
                // true->interleaved, onBuffer->audio stream callback
                XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                // final initialization params with default buffer size
                XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                // run stream
                // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                XtStream stream = device.OpenStream(deviceParams, null);
                var safeBuffer = XtSafeBuffer.Register(stream, true);
                // max frames to enter onBuffer * channels * bytes per sample
                this.BUFFER = new byte[stream.GetFrames() * 2 * 2];
                stream.Start();
                Thread.Sleep(1000000000);
            }
        }
    }

    // Update is called once per frame
    void Update()
    {

    }
}

`

[sjoerdvankreel]

Might be something unity specific. What OS and hardware are you running? Are you able to run the c# demo app (https://sjoerdvankreel.github.io/xt-audio/)?

[GDjkhp]

i use windows, and i think i dunno how to reference the dll in vs/unity, how do i make unity recognize the dll? i found some files like a config or something, and it links it to libxtaudio.so x86 x64 or something

[GDjkhp]

ok i somehow managed to import "managed" (no pun intended) and "native" dlls on unity, but, this poped out it may be my code above and it throws invalid operation exception at line XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);

[sjoerdvankreel]

Looks like you're trying to initialize xtaudio twice? Probably start is called more then once. Also note you never dispose of XtPlatform, XtDeviceList, XtDevice and XtStream. These should be in try/finally or using() blocks.

[GDjkhp]

i modified the code with using blocks, still the same error ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour
{
    // intermediate buffer
    public byte[] BUFFER = new byte[2048];

    // audio streaming callback
    public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
    {
        XtSafeBuffer safe = XtSafeBuffer.Get(stream);
        // lock buffer from native into java
        safe.Lock(buffer);
        // short[] because we specified INT16 below
        // this is the captured audio data
        short[] audio = (short[])safe.GetInput();
        // you want a spectrum analyzer, i dump to a file
        // but actually never dump to a file in any serious app
        // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
        processAudio(audio, buffer.frames);
        // unlock buffer from java into native
        safe.Unlock(buffer);
        return 0;
    }

    void processAudio(short[] audio, int frames)
    {
        // convert from short[] to byte[]
        for (int frame = 0; frame < frames; frame++)
        {
            // for 2 channels
            for (int channel = 0; channel < 2; channel++)
            {
                // 2 = channels again
                int sampleIndex = frame * 2 + channel;
                // 2 = 2 bytes for each short
                int byteIndex0 = sampleIndex * 2;
                int byteIndex1 = sampleIndex * 2 + 1;
                // probably some library method for this, somewhere
                // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
                BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
                BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

                /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
                pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                        / 32767.0f;*/
            }
        }

        // by now BYTES contains the data you want,
        // but be sure to account for frame count
        // (i.e. not all off BYTES may contain useful data,
        // might be some unused garbage at the end)

        // compute total bytes this round
        // = frame count * 2 channels * 2 bytes per short (INT16)
        int byteCount = frames * 2 * 2;
    }

    // Start is called before the first frame update
    void Start()
    {
        // this initializes platform dependent stuff like COM
        using(XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)) {
            // works on windows only, obviously
            XtService service = platform.GetService(XtSystem.WASAPI);
            // list input devices (this includes loopback)
            using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
            {
                for (int i = 0; i < list.GetCount(); i++)
                {
                    String deviceId = list.GetId(i);
                    XtDeviceCaps caps = list.GetCapabilities(deviceId);
                    // filter loopback devices
                    if (caps.Equals(XtDeviceCaps.Loopback))
                    {
                        String deviceName = list.GetName(deviceId);
                        // open device
                        using (XtDevice device = service.OpenDevice(deviceId))
                        {
                            // 16 bit 48khz
                            XtMix mix = new XtMix(48000, XtSample.Int16);
                            // 2 channels input, no masking
                            XtChannels channels = new XtChannels(2, 0, 0, 0);
                            // final audio format
                            XtFormat format = new XtFormat(mix, channels);
                            // query min/max/default buffer sizes
                            XtBufferSize bufferSize = device.GetBufferSize(format);
                            // true->interleaved, onBuffer->audio stream callback
                            XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                            // final initialization params with default buffer size
                            XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                            // run stream
                            // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                            using (XtStream stream = device.OpenStream(deviceParams, null))
                            {
                                var safeBuffer = XtSafeBuffer.Register(stream, true);
                                // max frames to enter onBuffer * channels * bytes per sample
                                BUFFER = new byte[stream.GetFrames() * 2 * 2];
                                // run for 1 second
                                stream.Start();
                                Thread.Sleep(1000000000);
                            }
                        }
                    }
                }
            }
        }
    }

    // Update is called once per frame
    void Update()
    {

    }
}

`

[sjoerdvankreel]

You're trying to initialize the XtAudio library twice. Is Start() by any chance running on multiple threads? The assertion failure you're seeing is from row 59 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtAudio.cpp, which is cleaned up by the using block on row 9 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtPlatform.cpp.

[GDjkhp]

Start() only run once, what should i change so i only initialize xt audio once?

is it this line?

XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)

not to be confused on XtAudio (my class) to Xt.XtAudio (library)

should i rename my script?

[sjoerdvankreel]

This got to be related to the way unity operates. What happens if you just run one of the sample c# programs? https://sjoerdvankreel.github.io/xt-audio/

[GDjkhp]

running playback demo for .net gives me the same error at using (XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero))

[sjoerdvankreel]

Is that with or without unity in the mix? Or just the plain playback demo? Can you post full source code? Also, what happens when you run the pre-compiled demo app (https://sjoerdvankreel.github.io/xt-audio/dist/xt-audio-1.9.zip)?

[GDjkhp]

i just copy and pasted the code here: https://sjoerdvankreel.github.io/xt-audio/#simple-playback-net

full source code (change txt to cs, then paste this on a unity project)

XtAudio.txt

unity compiles scripts on source, i'm kinda new to unity

[sjoerdvankreel]

Sorry man I don't know the first thing about unity. Does the playback sample run as expected without unity? I.e. in a standard console app? If so, I'm closing this because it has to be unity related.

[GDjkhp]

i kinda figured it out, by disabling unity audio, and removing the for loop

for (int i = 0; i < list.GetCount(); i++)

putting zero at

String deviceId = list.GetId(0);

and removing this if statement

// filter loopback devices if (caps.Equals(XtDeviceCaps.Loopback))

so, new problem arises, unity only runs in 1 thread, is it possible to remove the Thread.Sleep and replacing it with a couroutine or something?

[GDjkhp]

running thread sleep freezes the entire unity editor…

[sjoerdvankreel]

What are you trying to accomplish anyway? Why not just use the built-in unity audio features?

[GDjkhp]

https://www.youtube.com/watch?v=sj3DOQJ4GvI&ab_channel=JohnKennedyPe%C3%B1a

it kinda works now, but it's lagging the entire editor, also it only works 1 time, and it's because unity doesn't dispose xt audio properly, look at the values of BUFFER array carefully

[sjoerdvankreel]

Do you still have Thread.Sleep in there? If so thats probably why it's breaking the editor. About the dispose call, it seems like Start() is only called once for the lifetime of your unity script (just glanced over it quickly, so i may be wrong). You'll have to find a way of starting/stopping the audio thread in a manner that's unity-friendly. There's stuff like Awake()/OnEnabled()/OnDisabled()/OnDestroy() in there that looks promising: https://gamedevbeginner.com/start-vs-awake-in-unity/.

[GDjkhp]

i'm using a coroutine instead of Thread.Sleep()

[GDjkhp]

i'm going to write something on OnDestroy() to destroy xtplatform

[GDjkhp]

thanks man, now it works!

i modified the code to destroy xtplatform on OnDestroy(), now i can run xtaudio multiple times, the lagging problem was fixed by NOT letting the editor read/write the BUFFER array (by collapsing/hiding the component from the inspector), you can now close this

[GDjkhp]

i noticed the buffer array doesn't return negative values, only positive values

[sjoerdvankreel]

That's expected as byte is an unsigned type. It's just a raw memory view of the incoming short[] array, which does contain negative values.

[GDjkhp]

how do i turn it into a signed pcm data? do i have to change something in this line of code?

BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8); BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

[sjoerdvankreel]

• short[] audio = (short[])safe.GetInput(); That's already signed 16 bit pcm because you did:
• XtMix mix = new XtMix(48000, XtSample.Int16); Also, BUFFER[] is also signed 16 bit pcm, just represented using a byte[]. You could just as easily store it in a sbyte[] (signed byte array), would still be 16 bit pcm. The only way to get truly unsigned audio is using XtSample.UInt8.

[GDjkhp]

woah, i didn't know sbyte was a thing, here's the working code ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour {

// intermediate buffer
public sbyte[] BUFFER;

// Start is called before the first frame update
void Start()
{
    StartCoroutine("mainMain");
}

// Update is called once per frame
void Update()
{

}

// audio streaming callback
public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
{
    XtSafeBuffer safe = XtSafeBuffer.Get(stream);
    // lock buffer from native into java
    safe.Lock(buffer);
    // short[] because we specified INT16 below
    // this is the captured audio data
    short[] audio = (short[])safe.GetInput();
    // you want a spectrum analyzer, i dump to a file
    // but actually never dump to a file in any serious app
    // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
    processAudio(audio, buffer.frames);
    // unlock buffer from java into native
    safe.Unlock(buffer);
    return 0;
}

void processAudio(short[] audio, int frames)
{
    // convert from short[] to byte[]
    for (int frame = 0; frame < frames; frame++)
    {
        // for 2 channels
        for (int channel = 0; channel < 2; channel++)
        {
            // 2 = channels again
            int sampleIndex = frame * 2 + channel;
            // 2 = 2 bytes for each short
            int byteIndex0 = sampleIndex * 2;
            int byteIndex1 = sampleIndex * 2 + 1;
            // probably some library method for this, somewhere
            // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
            BUFFER[byteIndex0] = (sbyte)((audio[sampleIndex] & 0x000000FF) >> 8);
            BUFFER[byteIndex1] = (sbyte)((audio[sampleIndex] & 0x0000FF00) >> 8);

            /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
            pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                    / 32767.0f;*/
        }
    }

    // by now BYTES contains the data you want,
    // but be sure to account for frame count
    // (i.e. not all off BYTES may contain useful data,
    // might be some unused garbage at the end)

    // compute total bytes this round
    // = frame count * 2 channels * 2 bytes per short (INT16)
    int byteCount = frames * 2 * 2;
}

XtPlatform platform;

private void OnDestroy()
{
    platform.Dispose();
}

public IEnumerator mainMain()
{
    // this initializes platform dependent stuff like COM
    using (platform = Xt.XtAudio.Init(null, IntPtr.Zero))
    {
        // works on windows only, obviously
        XtService service = platform.GetService(XtSystem.WASAPI);
        // list input devices (this includes loopback)
        using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
        {
            for (int i = 0; i < list.GetCount(); i++)
            {
                String deviceId = list.GetId(i);
                XtDeviceCaps caps = list.GetCapabilities(deviceId);
                // filter loopback devices
                if (caps.HasFlag(XtDeviceCaps.Loopback))
                {
                    String deviceName = list.GetName(deviceId);
                    Debug.Log(deviceName);
                    // open device
                    using (XtDevice device = service.OpenDevice(deviceId))
                    {
                        // 16 bit 48khz
                        XtMix mix = new XtMix(48000, XtSample.Int16);
                        // 2 channels input, no masking
                        XtChannels channels = new XtChannels(2, 0, 0, 0);
                        // final audio format
                        XtFormat format = new XtFormat(mix, channels);
                        // query min/max/default buffer sizes
                        XtBufferSize bufferSize = device.GetBufferSize(format);
                        // true->interleaved, onBuffer->audio stream callback
                        XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                        // final initialization params with default buffer size
                        XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                        // run stream
                        // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                        using (XtStream stream = device.OpenStream(deviceParams, null))
                        {
                            var safeBuffer = XtSafeBuffer.Register(stream, true);
                            // max frames to enter onBuffer * channels * bytes per sample
                            BUFFER = new sbyte[stream.GetFrames() * 2 * 2];
                            // run for 1 second
                            stream.Start();
                            yield return new WaitForSeconds(1000000000);
                            // Thread.Sleep(1000000000);
                        }
                    }
                }
            }
        }
    }  
}

}

`

video link

thanks for the help man, really appreciate it, you can now close this for real

uhm one more thing, can you build xt-audio for java 8? i tried porting the java source to work with java 8 but it gives me weird errors while running the compiled jar binary... also are you planning to release xt audio on the unity asset store?

[sjoerdvankreel]

Hey, java 9 support was added in v1.6 to support the new java module system, but I didn't expect that to break pre-v9 support. What exactly isnt working? Also no, xtaudio is available on nuget for .net and mvn central for java, and source code and/or precompiled binaries for c and c++. I don't plan to support any more delivery channels, sorry.

[GDjkhp]

oh i fixed it, i found out I'm using an old version, because of the natives "xt-core" was changed to "xt-audio", and i successfully ported it on java 8...