Potential for Audio support

[Soaryn]

Partially curious how difficult it would be to add the ability route audio into and out of Unity via NDI.

For case scenario, I am attempting to use unity as a scene composer. The goal is to have Multiple NDI video channels come in and be 'mixed' together in Unity in a scene composition, then output a couple NDI outputs to the network. The problem, is that on the video output at the moment, their is no way to send a combined Audio along with the video.

The hope, is to help people who are on a budget achieve something close to or surpassing that of a TriCaster system without having to change their setup too elaborately. Using Unity as a scene composer seems rather viable.

[keijiro]

It's technically possible. Implementing it is moderately difficult (requiring a few man weeks). Making it shippable is quite difficult (requiring another few man weeks and lots of testing). I'm interested in it, but I have some to-do items before it (receiver UI improvement, interlace support, frame synchronization support, timecode support, latency improvement, etc.).

I'll keep this issue opened, but please don't expect that it'll be done in the near future.

[wsteelenyc]

Just curious if the update to the NDI SDK 4 enables audio in your plugin? My other question would be what benefits are there now that version 4 of the SDK is being used.

[keijiro]

Just curious if the update to the NDI SDK 4 enables audio in your plugin?

No, nothing is related to this issue.

My other question would be what benefits are there now that version 4 of the SDK is being used.

I was just asked to update, so I did. I don't think there is a benefit, but for me, just updating it is far easier than arguing.

[SSormunen]

I have to say that i really appreciate that you @keijiro continue to update this plugin with latest NDI binaries. Big thanx for that!

Supporting latest binaries actually brings lot of benefits. The actual NDI encoding is improving in every release, and if sender application is capable of sending 4.1, and receiver can receive 4.1, we save bandwidth. Of course it also allows to implement new features that it supports.

This plugin is currently only actually easy to use plugin for NDI in Unity and i use it a lot. Audio support is also something that i also miss a lot. Video and audio go hand in hand in many of my projects.

And I think that adding it, would definetly boost peoples interest into this plugin. I truly hope that you find some time to think about adding it. Or we could find joint effort and help to get it added. I know that it's not your main priority and i'm not pushing.. just hoping.. :) Maybe releasing version with audio support in Asset store, with price tag of example 20$, could be one option to compensate the time spend on adding it.. ;) I would be more than happy to pay for that feature. Basic "video only" version could still be available for free.

[Harvey3141]

+1 for Audio support

[sturbefield]

We also would really like to see audio supported, we have a use case for the receiver to view a broadcast stream that contains audio and video and we would like to be able to render and hear the content within the Unity scene.

[wsteelenyc]

If you came up with an amount I'm sure there is a number of people (including me) willing to pay for /fund the audio support development!

[keijiro]

Please don't expect this happens in the near future. Audio support is quite difficult to implement without introducing audio crackling/stutters. If you're okay with these kinds of audio glitches, it's not too difficult, but I don't think it's a practical option.

[wsteelenyc]

@keijiro If you got it started there's a lot of innovated people including myself to work out the kinks. Where to get started is usually the hardest for me. Once there's a starting point I am usually pretty good debugging an implementing fixes. I also have a direct line of contact Dr. Cross, so could leverage that if we at least get started with audio even if not perfect.

[velenoise]

I'm currently giving it a go as I need this functionality, and getting some progress, but I'm having a hard time debugging the C++ side. @keijiro do you have any inputs you could share on how to debug the native side (in windows specifically)?

[keijiro]

@velenoise I think you can simply attach the debugger to the running Unity Editor process.

[velenoise]

@keijiro thanks for the reply, and sorry for the delayed response, got caught up in other stuff. I did try attaching the debugger, but it didn't change anything when unity crashed. Also, I'm not sure if for that I should be using the debug version of the plugin, which I tried, but then unity would crash at play, probably because of some unmet dependency (the debug dll is considerably smaller than the release one). Any pointers on that?

[keijiro]

That's weird. If I remember correctly, it worked without any hassle.

Recently, I migrated to the MinGW/WSL2 cross compiling environment, so I haven't installed Visual Studio on my machine. Please ask other experts about the debugging method.

[velenoise]

Cool, I'll try to dig deeper into it. Thanks!

[enquel]

@keijiro @velenoise is there a chance for a fork with even basic audio support?

[velenoise]

@enquel you can find a basic send implementation in my fork. Bear in mind that it's pretty barebones currently: the NDI Sender component must live in the same gameobject as your Audio Listener. I've also only built the plugin for windows, as I have no other OS in hands, but it should be straightforward to compile and get it working.

@keijiro one question: it seems that the source code from the plugin disapeared from your repo's master branch, is that on purpose?

[keijiro]

@velenoise

one question: it seems that the source code from the plugin disapeared from your repo's master branch, is that on purpose?

Yes because the new version doesn't use any C++ code. Now it's only implemented using C# interop.

[enquel]

@velenoise thank you very much! No worries, I will modify it if needed, at least now I have a starting point. Much obliged!

[velenoise]

@keijiro

Yes because the new version doesn't use any C++ code. Now it's only implemented using C# interop.

Ahh, got it. I was taking a look at the new source, it seems a lot clearer. Testing my initial implementation further, it seems to be a little unstable yet, I'll try rewriting it using the interop, and if it works I'll open a pull request for it.

[velenoise]

Audio sender is implemented in my fork using the new interop method. A lot more stable as well. @enquel I think you'd rather use the latest commit for whatever you're doing, it's a lot more stable.

[enquel]

Audio sender is implemented in my fork using the new interop method. A lot more stable as well. @enquel I think you'd rather use the latest commit for whatever you're doing, it's a lot more stable.

@velenoise thank you very much, have a good night.

[benwager]

Current implementation calls NDIlib_recv_capture_v2 in an Update Loop, which makes sense when you are just capturing VideoFrames. But this method also returns other frame types (Metadata / AudioFrame etc) so you need to call it way more frequently to get everything. Really you have to call it on a separate thread.

Also, the incoming AudioFrame comes in as planar, which needs to be converted. There's a convenient method for this, NDIlib_util_audio_to_interleaved_32f_v2. And then, you need to convert the byte array that produces to floats, which you can finally push into an audiosource via OnAudioFilterRead.

[wsteelenyc]

@benwager Did you go any further or start a fork for receiving audio? I'm getting back into myself and have since done a lot with the android NDI SDK's so maybe I can help. I'd love to get audio support working!

[darktable]

@keijiro (and anyone else interested), I have a branch with an audio receiver almost working. I've been doing most of my testing with a stereo source and sync looks good, but a bit of latency behind video. The four channel source generated by the NDI Test Pattern generator has more issues. I think I may need to discard the extra channel data. I don't know how Unity handles four channel audio clips.

This involved a large refactor of NdiReceiver.cs that may not match your style. I'm hoping someone who's worked with audio more than me can help address issues. https://github.com/darktable/KlakNDI/tree/audio-receiver

[wsteelenyc]

@darktable I will take a look. I'm pretty good at debugging and increasing performance. Just to clarify your "audio-reciever" fork is for both audio and video, correct?

[Ste-RH]

Thanks for your work on adding in audio support @darktable, and thanks @keijiro as ever for this fantastic resource and yout continued support for it. It really is top notch.

Have had a need for NDI + audio on a project and so have taken your work and made some modification to limit audio glitches/gapping that were evident.

We have put them in the audio-performance-improvements branch, and added a pull request to the audio branch here.

Essentially this comes down to not blocking the Unity audio thread in the grab audio callback. We have seen this before in one of our commercially available products. For that we implemented a lock-free circular buffer (in c/c++) as we were seeing locking of even a modest time causing issues. We have not eradicated the lock completely in these changes, but we have reduced it right back to pretty much single or double array copies.

Another change we made was to not touch the unity audio buffer unless there are enough samples to fill. Taking this further, we actually store a number of read ahead sample frames so that different timings of the threads and their overlapping does not cause an underrun of audio frames available.

Further advances here would be:

• make the circular buffer completely lock-free (not dedicated much brain power to this, but not sure how we could do this just in c# though. Thoughts anyone?)
• Add in a resampler unit for differences in samplerate of the NDI stream coming in and what the Unity project audio sample rate is set to. We have some experience in this and might be forced to implement a native c/c++ plugin to accomplish this in the coming weeks/months.
• Add in some channel mapping to again couple what comes over the NDI feed and what Unity is expecting. As with resampling, this might be on the cards for us as a requirement.

Naturally we will feed the above back into KlakNDI if we end up having to implement them.

[owen109]

@RenderHeadsSte I'm having some problems implementing your branch. I dragged the audio source into the slot and hit play a couple debug messages come through but no audio playback. tested the NDI with NDI studio monitor and I hear the audio on there so there definitely should be audio coming through. then the second time I hit play I Immediately get an error about a memory leak "A Native Collection has not been disposed, resulting in a memory leak. Enable Full StackTraces to get more details." any ideas on how I would solve this? been messing around with it for about 3 hours and haven't come to any solutions.

[krisrok]

@darktable @RenderHeadsSte thanks for your work! I'm running into the same problems as @owen109 describes. Any prerequisites we'd have to know about?

[Ste-RH]

Sorry for the super slow response @owen109. I was only alerted to it when @krisrok posted today.

Re the memory leak...this is likely to be 'm_aTempAudioPullBuffer' in NDIReceive.cs not being cleaned up. Suggest clean up in OnDestroy() in the same file.

The lack of audio is probably down to a mismatch between the NDI source and what the Unity audio thread is expecting. You have to ensure they are aligned in terms of sample rate, and channels. Check what your source is, and ensure that Unity and your hardware are set to the same. In our use-case the NDI feed is coming in from a Birddog box which ensures a 48KHz stereo audio stream.

Hope this helps!

[krisrok]

Regarding the missing audio: Thanks for the valuable advice @RenderHeadsSte. I still have no luck with it though.

Verified my setup:

• Audio device is set to 48khz, stereo, 16bit
• Unity's AudioSettings.outputSampleRate and .speakerMode are set to 48000 and stereo
• OBS output is set to 48khz, stereo
• VLC codec info says 48khz, stereo, 32bit
• Unity logs PrepareAudioSource: Creating audio clip to match frame data: AudioFrame: SampleRate: 48000 Channels: 2

Observations:

• No sound from either source (OBS nor VLC) in Unity
• OBS' audio meters move (e.g. when using VLC as input)
• NDI Studio Monitor plays audio as expected
• When selecting the runtime created AudioClip in Unity's inspector it is just a flat line (though i don't know if this preview would even work)
• When breaking in NdiReceiver.FillAudioBuffer and inspecting m_aTempSamplesArray i can see there's variation in the samples

I wonder where it goes wrong.

[krisrok]

Ah, got it. The NdiReceiver.OnAudioFilterRead is the method actually providing the samples to Unity's audio processing. The detour via AudioClip I guess is a remnant of the previous implementation.

NdiReceiver.OnAudioFilterRead is one of Unity's automagically-called methods and needs an AudioSource component on the same GameObject. My AudioSource was on a different GameObject before.

Now there's audio coming in! But super choppy. At least some progress.

[Ste-RH]

Nice one! Am sure that gotcha will help others :)

Your choppy audio could be 16/32 bit difference?

[krisrok]

Luckily the choppiness came from my tinkering in search of the problem. Reversed those changes and all is good now :)

Thanks for your help!

[Ste-RH]

Awesome! Would love to find some time to add in a resampler and channel mapping so the solution is as complete as we can make it. Just...time! :/

[krisrok]

Yeah resampling would be a great addition. Right now in a software-only setup it works by offloading this to OBS. It introduces some delay but still, it works without additional hardware which is always nice for hobbyists.

[krisrok]

I've pushed some additional improvements to my branch audio-receiver-improvements based on the work already done by others here.

Please note it only affects the receiving parts.

Short outline:

• Minor improvements to the custom editor
• Remove remnants of the old route via AudioClip
• Clear buffers to get rid of the A Native Collection has not been disposed error
• Remove some debug logging
• Add ways for using an AudioSource living on another GameObject (via a bridging class)
• Using memcpy if sample rate and channel count matches
• Fallbacks if they do not match:
  • Resample data if needed
  • Mix all input channels and write them to all output channels

It'd be great if someone could review my changes so maybe one day this whole thing will find its way into @keijiro's repo :)

Some other random thoughts:

• A "real" channel remapping would be great but I'm lacking knowledge
• I mainly used VLC as NDI source and playing around with different media formats often results in desynchronization of the video and audio frames in Unity.
  • Is there some way to "resync"?
  • Can we introduce a fixed audio delay?
• Received audio is never rendered spatially because of OnAudioFilterRead() -- there are workarounds though

[TheXRMonk]

Audio sender is implemented in my fork using the new interop method. A lot more stable as well. @enquel I think you'd rather use the latest commit for whatever you're doing, it's a lot more stable.

Is this mono only? I cant seem to get stereo through.

[SavvaVorobev]

Hey guys, I've downloaded @krisrok version of plugin and achieved audio from camera once, when i've changed audio source in play mode to the one attached to gameobj with Ndi reciever . After that I wasn't able to get sound no matter what I did. While debugging i got this in console "received StatusChange: VideoFrame: [0,0] Format: 0 FrameRate: не число Aspect: 0 Timecode: 0 Timestamp: 0 AudioFrame: SampleRate: 0 Channels: 0 Samples: 0 Timecode: 0 ChannelStride: 0 Timestamp: 0 MetadataFrame: Length: 0 Timecode: 0 Data: " Can smbd suggest what can i do to get stable audio from camera. (ps I use birddog 4k converter)

[brunoatvenosa]

Thanks to everyone that made audio possible! @krisrok master branch is working great on editor and with debug options enabled, but for windows builds the unity player crashes when I try to destroy the NdiReceiver. The last Debug.Log I can get from unity is before _converter?.Dispose();

void ReleaseInternalObjects()
{
  Debug.Log("here");
  _converter?.Dispose();
  _converter = null;
}

Anyone else experience this behavior?

Edit: Solution:

In NdiReceiver.Receiver

//change
ReleaseInternalObjects(); 
//for
mainThreadContext.Post(ReleaseInternalObjects, _converter);

and update ReleaseInternalObjects to match signature with parameter initialized with null

[tomsuperman]

@krisrok version still works but you need to fixed the issue which mostly introduced by Unity latest version. I did manage to have Sender through Windows or Mac (OBS), and the audio can be received by Android (SDK 31-34), iPhone (version 15 to 17.4). However, the quality is ok with some minor choppy effects, and the delay is around 200ms. It would be nice to have real channel remapping as krisrok mentioned.