Closed GDjkhp closed 3 years ago
Not sure what you mean? You can probably use them side by side though.
i mean i wanna use it in c# programming language, just like java
Sure that's possible, xtaudio has .net bindings. Supports both .net framework and core. See nuget package here: https://www.nuget.org/packages/Xt.Audio/. Just keep in mind supported backends are x86/64 windows and linux only, whereas unity looks like it supports a lot more platforms.
it gives me an error, dll not found
I installed it on NuGetForUnity plugin
and then I ported your code
`
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Xt;
using System;
using System.Threading;
public class XtAudio : MonoBehaviour
{
// intermediate buffer
public byte[] BUFFER = new byte[2048];
// audio streaming callback
public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
{
XtSafeBuffer safe = XtSafeBuffer.Get(stream);
// lock buffer from native into java
safe.Lock(buffer) ;
// short[] because we specified INT16 below
// this is the captured audio data
short[] audio = (short[])safe.GetInput();
// you want a spectrum analyzer, i dump to a file
// but actually never dump to a file in any serious app
// see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
processAudio(audio, buffer.frames);
// unlock buffer from java into native
safe.Unlock(buffer);
return 0;
}
void processAudio(short[] audio, int frames)
{
// convert from short[] to byte[]
for (int frame = 0; frame < frames; frame++)
{
// for 2 channels
for (int channel = 0; channel < 2; channel++)
{
// 2 = channels again
int sampleIndex = frame * 2 + channel;
// 2 = 2 bytes for each short
int byteIndex0 = sampleIndex * 2;
int byteIndex1 = sampleIndex * 2 + 1;
// probably some library method for this, somewhere
// TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);
/*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
/ 32767.0f;*/
}
}
// by now BYTES contains the data you want,
// but be sure to account for frame count
// (i.e. not all off BYTES may contain useful data,
// might be some unused garbage at the end)
// compute total bytes this round
// = frame count * 2 channels * 2 bytes per short (INT16)
int byteCount = frames * 2 * 2;
}
// Start is called before the first frame update
void Start()
{
// this initializes platform dependent stuff like COM
XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);
// works on windows only, obviously
XtService service = platform.GetService(XtSystem.WASAPI);
// list input devices (this includes loopback)
XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input);
for (int i = 0; i < list.GetCount(); i++)
{
String deviceId = list.GetId(i);
XtDeviceCaps caps = list.GetCapabilities(deviceId);
// filter loopback devices
if (caps.Equals(XtDeviceCaps.Loopback))
{
String deviceName = list.GetName(deviceId);
// open device
XtDevice device = service.OpenDevice(deviceId);
// 16 bit 48khz
XtMix mix = new XtMix(48000, XtSample.Int16);
// Structs.XtMix mix = new Structs.XtMix(44100, Enums.XtSample.INT16);
// 2 channels input, no masking
XtChannels channels = new XtChannels(2, 0, 0, 0);
// final audio format
XtFormat format = new XtFormat(mix, channels);
// query min/max/default buffer sizes
XtBufferSize bufferSize = device.GetBufferSize(format);
// true->interleaved, onBuffer->audio stream callback
XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
// final initialization params with default buffer size
XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
// run stream
// safe buffer allows you to get java short[] instead on jna Pointer in the callback
XtStream stream = device.OpenStream(deviceParams, null);
var safeBuffer = XtSafeBuffer.Register(stream, true);
// max frames to enter onBuffer * channels * bytes per sample
this.BUFFER = new byte[stream.GetFrames() * 2 * 2];
stream.Start();
Thread.Sleep(1000000000);
}
}
}
// Update is called once per frame
void Update()
{
}
}
`
Might be something unity specific. What OS and hardware are you running? Are you able to run the c# demo app (https://sjoerdvankreel.github.io/xt-audio/)?
i use windows, and i think i dunno how to reference the dll in vs/unity, how do i make unity recognize the dll? i found some files like a config or something, and it links it to libxtaudio.so x86 x64 or something
ok i somehow managed to import "managed" (no pun intended) and "native" dlls on unity, but, this poped out
it may be my code above and it throws invalid operation exception at line
XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero);
Looks like you're trying to initialize xtaudio twice? Probably start is called more then once. Also note you never dispose of XtPlatform, XtDeviceList, XtDevice and XtStream. These should be in try/finally or using() blocks.
i modified the code with using blocks, still the same error ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;
public class XtAudio : MonoBehaviour
{
// intermediate buffer
public byte[] BUFFER = new byte[2048];
// audio streaming callback
public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
{
XtSafeBuffer safe = XtSafeBuffer.Get(stream);
// lock buffer from native into java
safe.Lock(buffer);
// short[] because we specified INT16 below
// this is the captured audio data
short[] audio = (short[])safe.GetInput();
// you want a spectrum analyzer, i dump to a file
// but actually never dump to a file in any serious app
// see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
processAudio(audio, buffer.frames);
// unlock buffer from java into native
safe.Unlock(buffer);
return 0;
}
void processAudio(short[] audio, int frames)
{
// convert from short[] to byte[]
for (int frame = 0; frame < frames; frame++)
{
// for 2 channels
for (int channel = 0; channel < 2; channel++)
{
// 2 = channels again
int sampleIndex = frame * 2 + channel;
// 2 = 2 bytes for each short
int byteIndex0 = sampleIndex * 2;
int byteIndex1 = sampleIndex * 2 + 1;
// probably some library method for this, somewhere
// TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8);
BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);
/*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
/ 32767.0f;*/
}
}
// by now BYTES contains the data you want,
// but be sure to account for frame count
// (i.e. not all off BYTES may contain useful data,
// might be some unused garbage at the end)
// compute total bytes this round
// = frame count * 2 channels * 2 bytes per short (INT16)
int byteCount = frames * 2 * 2;
}
// Start is called before the first frame update
void Start()
{
// this initializes platform dependent stuff like COM
using(XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)) {
// works on windows only, obviously
XtService service = platform.GetService(XtSystem.WASAPI);
// list input devices (this includes loopback)
using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
{
for (int i = 0; i < list.GetCount(); i++)
{
String deviceId = list.GetId(i);
XtDeviceCaps caps = list.GetCapabilities(deviceId);
// filter loopback devices
if (caps.Equals(XtDeviceCaps.Loopback))
{
String deviceName = list.GetName(deviceId);
// open device
using (XtDevice device = service.OpenDevice(deviceId))
{
// 16 bit 48khz
XtMix mix = new XtMix(48000, XtSample.Int16);
// 2 channels input, no masking
XtChannels channels = new XtChannels(2, 0, 0, 0);
// final audio format
XtFormat format = new XtFormat(mix, channels);
// query min/max/default buffer sizes
XtBufferSize bufferSize = device.GetBufferSize(format);
// true->interleaved, onBuffer->audio stream callback
XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
// final initialization params with default buffer size
XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
// run stream
// safe buffer allows you to get java short[] instead on jna Pointer in the callback
using (XtStream stream = device.OpenStream(deviceParams, null))
{
var safeBuffer = XtSafeBuffer.Register(stream, true);
// max frames to enter onBuffer * channels * bytes per sample
BUFFER = new byte[stream.GetFrames() * 2 * 2];
// run for 1 second
stream.Start();
Thread.Sleep(1000000000);
}
}
}
}
}
}
}
// Update is called once per frame
void Update()
{
}
}
`
You're trying to initialize the XtAudio library twice. Is Start() by any chance running on multiple threads? The assertion failure you're seeing is from row 59 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtAudio.cpp, which is cleaned up by the using block on row 9 here: https://github.com/sjoerdvankreel/xt-audio/blob/master/src/core/xt/xt/api/XtPlatform.cpp.
Start() only run once, what should i change so i only initialize xt audio once?
is it this line?
XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero)
not to be confused on XtAudio (my class) to Xt.XtAudio (library)
should i rename my script?
This got to be related to the way unity operates. What happens if you just run one of the sample c# programs? https://sjoerdvankreel.github.io/xt-audio/
running playback demo for .net gives me the same error at
using (XtPlatform platform = Xt.XtAudio.Init(null, IntPtr.Zero))
Is that with or without unity in the mix? Or just the plain playback demo? Can you post full source code? Also, what happens when you run the pre-compiled demo app (https://sjoerdvankreel.github.io/xt-audio/dist/xt-audio-1.9.zip)?
i just copy and pasted the code here: https://sjoerdvankreel.github.io/xt-audio/#simple-playback-net
full source code (change txt to cs, then paste this on a unity project)
unity compiles scripts on source, i'm kinda new to unity
Sorry man I don't know the first thing about unity. Does the playback sample run as expected without unity? I.e. in a standard console app? If so, I'm closing this because it has to be unity related.
i kinda figured it out, by disabling unity audio, and removing the for loop
for (int i = 0; i < list.GetCount(); i++)
putting zero at
String deviceId = list.GetId(0);
and removing this if statement
// filter loopback devices if (caps.Equals(XtDeviceCaps.Loopback))
so, new problem arises, unity only runs in 1 thread, is it possible to remove the Thread.Sleep and replacing it with a couroutine or something?
running thread sleep freezes the entire unity editor…
What are you trying to accomplish anyway? Why not just use the built-in unity audio features?
https://www.youtube.com/watch?v=sj3DOQJ4GvI&ab_channel=JohnKennedyPe%C3%B1a
it kinda works now, but it's lagging the entire editor, also it only works 1 time, and it's because unity doesn't dispose xt audio properly, look at the values of BUFFER array carefully
Do you still have Thread.Sleep in there? If so thats probably why it's breaking the editor. About the dispose call, it seems like Start() is only called once for the lifetime of your unity script (just glanced over it quickly, so i may be wrong). You'll have to find a way of starting/stopping the audio thread in a manner that's unity-friendly. There's stuff like Awake()/OnEnabled()/OnDisabled()/OnDestroy() in there that looks promising: https://gamedevbeginner.com/start-vs-awake-in-unity/.
i'm using a coroutine instead of Thread.Sleep()
i'm going to write something on OnDestroy() to destroy xtplatform
thanks man, now it works!
i modified the code to destroy xtplatform on OnDestroy(), now i can run xtaudio multiple times, the lagging problem was fixed by NOT letting the editor read/write the BUFFER array (by collapsing/hiding the component from the inspector), you can now close this
i noticed the buffer array doesn't return negative values, only positive values
That's expected as byte is an unsigned type. It's just a raw memory view of the incoming short[] array, which does contain negative values.
how do i turn it into a signed pcm data? do i have to change something in this line of code?
BUFFER[byteIndex0] = (byte)((audio[sampleIndex] & 0x000000FF) >> 8); BUFFER[byteIndex1] = (byte)((audio[sampleIndex] & 0x0000FF00) >> 8);
woah, i didn't know sbyte was a thing, here's the working code ` using System.Collections; using System.Collections.Generic; using UnityEngine; using Xt; using System; using System.Threading;
public class XtAudio : MonoBehaviour {
// intermediate buffer
public sbyte[] BUFFER;
// Start is called before the first frame update
void Start()
{
StartCoroutine("mainMain");
}
// Update is called once per frame
void Update()
{
}
// audio streaming callback
public int onBuffer(XtStream stream, in XtBuffer buffer, object user)
{
XtSafeBuffer safe = XtSafeBuffer.Get(stream);
// lock buffer from native into java
safe.Lock(buffer);
// short[] because we specified INT16 below
// this is the captured audio data
short[] audio = (short[])safe.GetInput();
// you want a spectrum analyzer, i dump to a file
// but actually never dump to a file in any serious app
// see http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
processAudio(audio, buffer.frames);
// unlock buffer from java into native
safe.Unlock(buffer);
return 0;
}
void processAudio(short[] audio, int frames)
{
// convert from short[] to byte[]
for (int frame = 0; frame < frames; frame++)
{
// for 2 channels
for (int channel = 0; channel < 2; channel++)
{
// 2 = channels again
int sampleIndex = frame * 2 + channel;
// 2 = 2 bytes for each short
int byteIndex0 = sampleIndex * 2;
int byteIndex1 = sampleIndex * 2 + 1;
// probably some library method for this, somewhere
// TODO: implement audio consumer here, see KJDSPAudioDataConsumer.java line 406
BUFFER[byteIndex0] = (sbyte)((audio[sampleIndex] & 0x000000FF) >> 8);
BUFFER[byteIndex1] = (sbyte)((audio[sampleIndex] & 0x0000FF00) >> 8);
/*pLeftChannel[byteIndex0] = (float) (((int) BUFFER[sampleIndex + 1] << 8) + (BUFFER[sampleIndex] & 0xff)) / 32767.0f;
pRightChannel[byteIndex1] = (float) (((int) BUFFER[sampleIndex + 3] << 8) + (BUFFER[sampleIndex + 2] & 0xff))
/ 32767.0f;*/
}
}
// by now BYTES contains the data you want,
// but be sure to account for frame count
// (i.e. not all off BYTES may contain useful data,
// might be some unused garbage at the end)
// compute total bytes this round
// = frame count * 2 channels * 2 bytes per short (INT16)
int byteCount = frames * 2 * 2;
}
XtPlatform platform;
private void OnDestroy()
{
platform.Dispose();
}
public IEnumerator mainMain()
{
// this initializes platform dependent stuff like COM
using (platform = Xt.XtAudio.Init(null, IntPtr.Zero))
{
// works on windows only, obviously
XtService service = platform.GetService(XtSystem.WASAPI);
// list input devices (this includes loopback)
using (XtDeviceList list = service.OpenDeviceList(XtEnumFlags.Input))
{
for (int i = 0; i < list.GetCount(); i++)
{
String deviceId = list.GetId(i);
XtDeviceCaps caps = list.GetCapabilities(deviceId);
// filter loopback devices
if (caps.HasFlag(XtDeviceCaps.Loopback))
{
String deviceName = list.GetName(deviceId);
Debug.Log(deviceName);
// open device
using (XtDevice device = service.OpenDevice(deviceId))
{
// 16 bit 48khz
XtMix mix = new XtMix(48000, XtSample.Int16);
// 2 channels input, no masking
XtChannels channels = new XtChannels(2, 0, 0, 0);
// final audio format
XtFormat format = new XtFormat(mix, channels);
// query min/max/default buffer sizes
XtBufferSize bufferSize = device.GetBufferSize(format);
// true->interleaved, onBuffer->audio stream callback
XtStreamParams streamParams = new XtStreamParams(true, onBuffer, null, null);
// final initialization params with default buffer size
XtDeviceStreamParams deviceParams = new XtDeviceStreamParams(streamParams, format, bufferSize.current);
// run stream
// safe buffer allows you to get java short[] instead on jna Pointer in the callback
using (XtStream stream = device.OpenStream(deviceParams, null))
{
var safeBuffer = XtSafeBuffer.Register(stream, true);
// max frames to enter onBuffer * channels * bytes per sample
BUFFER = new sbyte[stream.GetFrames() * 2 * 2];
// run for 1 second
stream.Start();
yield return new WaitForSeconds(1000000000);
// Thread.Sleep(1000000000);
}
}
}
}
}
}
}
}
`
thanks for the help man, really appreciate it, you can now close this for real
uhm one more thing, can you build xt-audio for java 8? i tried porting the java source to work with java 8 but it gives me weird errors while running the compiled jar binary... also are you planning to release xt audio on the unity asset store?
Hey, java 9 support was added in v1.6 to support the new java module system, but I didn't expect that to break pre-v9 support. What exactly isnt working? Also no, xtaudio is available on nuget for .net and mvn central for java, and source code and/or precompiled binaries for c and c++. I don't plan to support any more delivery channels, sorry.
oh i fixed it, i found out I'm using an old version, because of the natives "xt-core" was changed to "xt-audio", and i successfully ported it on java 8...
i wanna try to implement xt-audio on unity3d, is it possible?