sjoerdvankreel / xt-audio

Platform independent low-latency audio for C, C++, Java and .NET.
https://sjoerdvankreel.github.io/xt-audio/
Other
64 stars 12 forks source link

C# language support #15

Closed GDjkhp closed 3 years ago

GDjkhp commented 3 years ago

i wanna try to implement xt-audio on unity3d, is it possible?

sjoerdvankreel commented 3 years ago

Not sure what you mean? You can probably use them side by side though.

GDjkhp commented 3 years ago

i mean i wanna use it in c# programming language, just like java

sjoerdvankreel commented 3 years ago

Sure that's possible, xtaudio has .net bindings. Supports both .net framework and core. See nuget package here: https://www.nuget.org/packages/Xt.Audio/. Just keep in mind supported backends are x86/64 windows and linux only, whereas unity looks like it supports a lot more platforms.

GDjkhp commented 3 years ago

it gives me an error, dll not found image

I installed it on NuGetForUnity plugin image

and then I ported your code

`
using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour
{
    // intermediate buffer
    public byte[] BUFFER = new byte[2048];

    // audio streaming callback
    public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
    {
        XtSafeBuffer safe = XtSafeBuffer.Get(stream);
        // lock buffer from native into java
        safe.Lock(buffer) ;
        // short[] because we specified INT16 below
        // this is the captured audio data
        short[] audio = (short[])safe.GetInput();
        // you want a spectrum analyzer, i dump to a file
        // but actually never dump to a file in any serious app
        // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
        processAudio(audio, buffer.frames);
        // unlock buffer from java into native
        safe.Unlock(buffer);
        return 0;
    }

    void processAudio(short[] audio, int frames)
    {
        // convert from short[] to byte[]
        for (int frame = 0; frame < frames; frame++)
        {
            // for 2 channels
            for (int channel = 0; channel < 2; channel++)
            {
                // 2 = channels again
                int sampleIndex = frame * 2 + channel;
                // 2 = 2 bytes for each short
                int byteIndex0 = sampleIndex * 2;
                int byteIndex1 = sampleIndex * 2 + 1;
                // probably some library method for this, somewhere
                // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
                BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
                BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

                /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
                pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                        / 32767.0f;*/
            }
        }

        // by now BYTES contains the data you want,
        // but be sure to account for frame count
        // (i.e. not all off BYTES may contain useful data,
        // might be some unused garbage at the end)

        // compute total bytes this round
        // = frame count * 2 channels * 2 bytes per short (INT16)
        int byteCount = frames * 2 * 2;
    }

    // Start is called before the first frame update
    void Start()
    {
        // this initializes platform dependent stuff like COM
        XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);
        // works on windows only, obviously
        XtService service = platform.GetService(XtSystem.WASAPI);
        // list input devices (this includes loopback)
        XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input);
        for (int i = 0; i < list.GetCount(); i++)
        {
            String deviceId = list.GetId(i);
            XtDeviceCaps caps = list.GetCapabilities(deviceId);
            // filter loopback devices
            if (caps.Equals(XtDeviceCaps.Loopback))
            {
                String deviceName = list.GetName(deviceId);
                // open device
                XtDevice device = service.OpenDevice(deviceId);
                // 16 bit 48khz
                XtMix mix = new XtMix(48000, XtSample.Int16);
                // Structs.XtMix mix = new Structs.XtMix(44100, Enums.XtSample.INT16);
                // 2 channels input, no masking
                XtChannels channels = new XtChannels(2, 0, 0, 0);
                // final audio format
                XtFormat format = new XtFormat(mix, channels);
                // query min/max/default buffer sizes
                XtBufferSize bufferSize = device.GetBufferSize(format);
                // true->interleaved, onBuffer->audio stream callback
                XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                // final initialization params with default buffer size
                XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                // run stream
                // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                XtStream stream = device.OpenStream(deviceParams, null);
                var safeBuffer = XtSafeBuffer.Register(stream, true);
                // max frames to enter onBuffer * channels * bytes per sample
                this.BUFFER = new byte[stream.GetFrames() * 2 * 2];
                stream.Start();
                Thread.Sleep(1000000000);
            }
        }
    }

    // Update is called once per frame
    void Update()
    {

    }
}

`

sjoerdvankreel commented 3 years ago

Might be something unity specific. What OS and hardware are you running? Are you able to run the c# demo app (https://sjoerdvankreel.github.io/xt-audio/)?

GDjkhp commented 3 years ago

i use windows, and i think i dunno how to reference the dll in vs/unity, how do i make unity recognize the dll? i found some files like a config or something, and it links it to libxtaudio.so x86 x64 or something

GDjkhp commented 3 years ago

ok i somehow managed to import "managed" (no pun intended) and "native" dlls on unity, but, this poped out image it may be my code above and it throws invalid operation exception at line XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);

sjoerdvankreel commented 3 years ago

Looks like you're trying to initialize xtaudio twice? Probably start is called more then once. Also note you never dispose of XtPlatform, XtDeviceList, XtDevice and XtStream. These should be in try/finally or using() blocks.

GDjkhp commented 3 years ago

i modified the code with using blocks, still the same error ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour
{
    // intermediate buffer
    public byte[] BUFFER = new byte[2048];

    // audio streaming callback
    public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
    {
        XtSafeBuffer safe = XtSafeBuffer.Get(stream);
        // lock buffer from native into java
        safe.Lock(buffer);
        // short[] because we specified INT16 below
        // this is the captured audio data
        short[] audio = (short[])safe.GetInput();
        // you want a spectrum analyzer, i dump to a file
        // but actually never dump to a file in any serious app
        // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
        processAudio(audio, buffer.frames);
        // unlock buffer from java into native
        safe.Unlock(buffer);
        return 0;
    }

    void processAudio(short[] audio, int frames)
    {
        // convert from short[] to byte[]
        for (int frame = 0; frame < frames; frame++)
        {
            // for 2 channels
            for (int channel = 0; channel < 2; channel++)
            {
                // 2 = channels again
                int sampleIndex = frame * 2 + channel;
                // 2 = 2 bytes for each short
                int byteIndex0 = sampleIndex * 2;
                int byteIndex1 = sampleIndex * 2 + 1;
                // probably some library method for this, somewhere
                // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
                BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
                BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

                /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
                pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                        / 32767.0f;*/
            }
        }

        // by now BYTES contains the data you want,
        // but be sure to account for frame count
        // (i.e. not all off BYTES may contain useful data,
        // might be some unused garbage at the end)

        // compute total bytes this round
        // = frame count * 2 channels * 2 bytes per short (INT16)
        int byteCount = frames * 2 * 2;
    }

    // Start is called before the first frame update
    void Start()
    {
        // this initializes platform dependent stuff like COM
        using(XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)) {
            // works on windows only, obviously
            XtService service = platform.GetService(XtSystem.WASAPI);
            // list input devices (this includes loopback)
            using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
            {
                for (int i = 0; i < list.GetCount(); i++)
                {
                    String deviceId = list.GetId(i);
                    XtDeviceCaps caps = list.GetCapabilities(deviceId);
                    // filter loopback devices
                    if (caps.Equals(XtDeviceCaps.Loopback))
                    {
                        String deviceName = list.GetName(deviceId);
                        // open device
                        using (XtDevice device = service.OpenDevice(deviceId))
                        {
                            // 16 bit 48khz
                            XtMix mix = new XtMix(48000, XtSample.Int16);
                            // 2 channels input, no masking
                            XtChannels channels = new XtChannels(2, 0, 0, 0);
                            // final audio format
                            XtFormat format = new XtFormat(mix, channels);
                            // query min/max/default buffer sizes
                            XtBufferSize bufferSize = device.GetBufferSize(format);
                            // true->interleaved, onBuffer->audio stream callback
                            XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                            // final initialization params with default buffer size
                            XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                            // run stream
                            // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                            using (XtStream stream = device.OpenStream(deviceParams, null))
                            {
                                var safeBuffer = XtSafeBuffer.Register(stream, true);
                                // max frames to enter onBuffer * channels * bytes per sample
                                BUFFER = new byte[stream.GetFrames() * 2 * 2];
                                // run for 1 second
                                stream.Start();
                                Thread.Sleep(1000000000);
                            }
                        }
                    }
                }
            }
        }
    }

    // Update is called once per frame
    void Update()
    {

    }
}

`

sjoerdvankreel commented 3 years ago

You're trying to initialize the XtAudio library twice. Is Start() by any chance running on multiple threads? The assertion failure you're seeing is from row 59 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtAudio.cpp, which is cleaned up by the using block on row 9 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtPlatform.cpp.

GDjkhp commented 3 years ago

Start() only run once, what should i change so i only initialize xt audio once?

is it this line?

XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)

not to be confused on XtAudio (my class) to Xt.XtAudio (library)

should i rename my script?

sjoerdvankreel commented 3 years ago

This got to be related to the way unity operates. What happens if you just run one of the sample c# programs? https://sjoerdvankreel.github.io/xt-audio/

GDjkhp commented 3 years ago

running playback demo for .net gives me the same error at using (XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero))

sjoerdvankreel commented 3 years ago

Is that with or without unity in the mix? Or just the plain playback demo? Can you post full source code? Also, what happens when you run the pre-compiled demo app (https://sjoerdvankreel.github.io/xt-audio/dist/xt-audio-1.9.zip)?

GDjkhp commented 3 years ago

i just copy and pasted the code here: https://sjoerdvankreel.github.io/xt-audio/#simple-playback-net

full source code (change txt to cs, then paste this on a unity project)

XtAudio.txt

unity compiles scripts on source, i'm kinda new to unity

sjoerdvankreel commented 3 years ago

Sorry man I don't know the first thing about unity. Does the playback sample run as expected without unity? I.e. in a standard console app? If so, I'm closing this because it has to be unity related.

GDjkhp commented 3 years ago

i kinda figured it out, by disabling unity audio, and removing the for loop

for (int i = 0; i < list.GetCount(); i++)

putting zero at

String deviceId = list.GetId(0);

and removing this if statement

// filter loopback devices if (caps.Equals(XtDeviceCaps.Loopback))

so, new problem arises, unity only runs in 1 thread, is it possible to remove the Thread.Sleep and replacing it with a couroutine or something?

it kinda works

GDjkhp commented 3 years ago

running thread sleep freezes the entire unity editor…

sjoerdvankreel commented 3 years ago

What are you trying to accomplish anyway? Why not just use the built-in unity audio features?

GDjkhp commented 3 years ago

https://www.youtube.com/watch?v=sj3DOQJ4GvI&ab_channel=JohnKennedyPe%C3%B1a

it kinda works now, but it's lagging the entire editor, also it only works 1 time, and it's because unity doesn't dispose xt audio properly, look at the values of BUFFER array carefully

sjoerdvankreel commented 3 years ago

Do you still have Thread.Sleep in there? If so thats probably why it's breaking the editor. About the dispose call, it seems like Start() is only called once for the lifetime of your unity script (just glanced over it quickly, so i may be wrong). You'll have to find a way of starting/stopping the audio thread in a manner that's unity-friendly. There's stuff like Awake()/OnEnabled()/OnDisabled()/OnDestroy() in there that looks promising: https://gamedevbeginner.com/start-vs-awake-in-unity/.

GDjkhp commented 3 years ago

i'm using a coroutine instead of Thread.Sleep()

GDjkhp commented 3 years ago

i'm going to write something on OnDestroy() to destroy xtplatform

GDjkhp commented 3 years ago

thanks man, now it works!

image

i modified the code to destroy xtplatform on OnDestroy(), now i can run xtaudio multiple times, the lagging problem was fixed by NOT letting the editor read/write the BUFFER array (by collapsing/hiding the component from the inspector), you can now close this

GDjkhp commented 3 years ago

i noticed the buffer array doesn't return negative values, only positive values

sjoerdvankreel commented 3 years ago

That's expected as byte is an unsigned type. It's just a raw memory view of the incoming short[] array, which does contain negative values.

GDjkhp commented 3 years ago

how do i turn it into a signed pcm data? do i have to change something in this line of code?

BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8); BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);

sjoerdvankreel commented 3 years ago
GDjkhp commented 3 years ago

woah, i didn't know sbyte was a thing, here's the working code ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;

public class XtAudio : MonoBehaviour {

// intermediate buffer
public sbyte[] BUFFER;

// Start is called before the first frame update
void Start()
{
    StartCoroutine("mainMain");
}

// Update is called once per frame
void Update()
{

}

// audio streaming callback
public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
{
    XtSafeBuffer safe = XtSafeBuffer.Get(stream);
    // lock buffer from native into java
    safe.Lock(buffer);
    // short[] because we specified INT16 below
    // this is the captured audio data
    short[] audio = (short[])safe.GetInput();
    // you want a spectrum analyzer, i dump to a file
    // but actually never dump to a file in any serious app
    // see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
    processAudio(audio, buffer.frames);
    // unlock buffer from java into native
    safe.Unlock(buffer);
    return 0;
}

void processAudio(short[] audio, int frames)
{
    // convert from short[] to byte[]
    for (int frame = 0; frame < frames; frame++)
    {
        // for 2 channels
        for (int channel = 0; channel < 2; channel++)
        {
            // 2 = channels again
            int sampleIndex = frame * 2 + channel;
            // 2 = 2 bytes for each short
            int byteIndex0 = sampleIndex * 2;
            int byteIndex1 = sampleIndex * 2 + 1;
            // probably some library method for this, somewhere
            // TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
            BUFFER[byteIndex0] = (sbyte)((audio[sampleIndex] & 0x000000FF) >> 8);
            BUFFER[byteIndex1] = (sbyte)((audio[sampleIndex] & 0x0000FF00) >> 8);

            /*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
            pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
                    / 32767.0f;*/
        }
    }

    // by now BYTES contains the data you want,
    // but be sure to account for frame count
    // (i.e. not all off BYTES may contain useful data,
    // might be some unused garbage at the end)

    // compute total bytes this round
    // = frame count * 2 channels * 2 bytes per short (INT16)
    int byteCount = frames * 2 * 2;
}

XtPlatform platform;

private void OnDestroy()
{
    platform.Dispose();
}

public IEnumerator mainMain()
{
    // this initializes platform dependent stuff like COM
    using (platform = Xt.XtAudio.Init(null, IntPtr.Zero))
    {
        // works on windows only, obviously
        XtService service = platform.GetService(XtSystem.WASAPI);
        // list input devices (this includes loopback)
        using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
        {
            for (int i = 0; i < list.GetCount(); i++)
            {
                String deviceId = list.GetId(i);
                XtDeviceCaps caps = list.GetCapabilities(deviceId);
                // filter loopback devices
                if (caps.HasFlag(XtDeviceCaps.Loopback))
                {
                    String deviceName = list.GetName(deviceId);
                    Debug.Log(deviceName);
                    // open device
                    using (XtDevice device = service.OpenDevice(deviceId))
                    {
                        // 16 bit 48khz
                        XtMix mix = new XtMix(48000, XtSample.Int16);
                        // 2 channels input, no masking
                        XtChannels channels = new XtChannels(2, 0, 0, 0);
                        // final audio format
                        XtFormat format = new XtFormat(mix, channels);
                        // query min/max/default buffer sizes
                        XtBufferSize bufferSize = device.GetBufferSize(format);
                        // true->interleaved, onBuffer->audio stream callback
                        XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
                        // final initialization params with default buffer size
                        XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
                        // run stream
                        // safe buffer allows you to get java short[] instead on jna Pointer in the callback
                        using (XtStream stream = device.OpenStream(deviceParams, null))
                        {
                            var safeBuffer = XtSafeBuffer.Register(stream, true);
                            // max frames to enter onBuffer * channels * bytes per sample
                            BUFFER = new sbyte[stream.GetFrames() * 2 * 2];
                            // run for 1 second
                            stream.Start();
                            yield return new WaitForSeconds(1000000000);
                            // Thread.Sleep(1000000000);
                        }
                    }
                }
            }
        }
    }  
}

}

`

xt audio ported successfully

video link

thanks for the help man, really appreciate it, you can now close this for real

uhm one more thing, can you build xt-audio for java 8? i tried porting the java source to work with java 8 but it gives me weird errors while running the compiled jar binary... also are you planning to release xt audio on the unity asset store?

sjoerdvankreel commented 3 years ago

Hey, java 9 support was added in v1.6 to support the new java module system, but I didn't expect that to break pre-v9 support. What exactly isnt working? Also no, xtaudio is available on nuget for .net and mvn central for java, and source code and/or precompiled binaries for c and c++. I don't plan to support any more delivery channels, sorry.

GDjkhp commented 3 years ago

oh i fixed it, i found out I'm using an old version, because of the natives "xt-core" was changed to "xt-audio", and i successfully ported it on java 8...