CRASHING with Windows Insider Build and Unity Body Tracking

[mcdopsa]

Describe the bug When initializing Azure Kinect Body Tracking in Unity, the entire editor will crash instantly. The Kinect RGB camera light flashes on, then remains off after the crash. The Kinect's IR projector turns on, and remains on after the crash. Opening Kinect Viewer v1.3.0, or Body Tracking Viewer executables work as expected on the same machine, with no issues running those applications on any of the various versions.

To Reproduce Since late fall 2019, all computers running 'Windows Insider Build' versions of Windows 10 consistently produce this issue. These computers were previously working with Unity + Azure Kinect Body Tracking, and developed this issue after a software update. With nearly a dozen computers tested, in various implementations, it remains consistent that only computers running Windows Insider Builds experience the error. This issue is pervasive on all versions of the Body Tracking SDK (tested 0.9.0 through 1.0.0), and all versions of Unity (tested 2018.3 through 2019.3) without exception.

Expected behavior Unity scene should initialize Kinect camera, and track body movements in editor on start, then turn off Kinect camera on exit.

Logs

Editor.log error.log

Screenshots

Desktop (please complete the following information):

• OS with Version: Windows 10 Pro, Version 2004, OS build 19041.21
• SDK Version: 0.9.0-1.0.0
• Firmware version: ALL

Additional context In addition to Microsoft's own Unity Samples (this repo), 3rd party implementations of Azure Kinect Body Tracking (like Rumen Filkov's examples, and other Open Source tools on Github) experience identical issues. We fear that as the latest Windows Insider Build becomes integrated into the mainstream Windows 10 pipeline, ALL UNITY-BASED AZURE KINECT APPLICATIONS MAY BE SUBJECT TO BREAKING. This is a big issue for us as developers, but would be an even larger issue for permanent installations, industrial or medical environments (existing and future) where preventing Windows 10 updates may not be possible or expected. If we can get to the bottom of this issue here, we can perhaps prevent ALL Azure Kinect developers having to retro-actively update their existing projects and installations!

[qm13]

Could you please file feedback to the Windows team using the Windows Feedback tool.

[yijiew]

It is a duplication of issue #932 . I have reopened the other issue and I will try to contact windows team for solution.

[mcdopsa]

It is a duplication of issue #932 . I have reopened the other issue and I will try to contact windows team for solution.

I am not sure that these issues are identical. To be clear, I am able to run the Body Tracking Viewer executable with no errors, my issues are specific to running within Unity editor and built Unity projects. Regardless there seems to be some issue with Windows insider + K4ABT, so thank you for reopening the issue and reaching out!

[mcdopsa]

Could you please file feedback to the Windows team using the Windows Feedback tool.

Can you please specify the tool you are referring to? A search returned 'Feedback Hub' and I am unsure if this is the 'Windows Feedback Tool' you mentioned. Happy to oblige, I just want to confirm I've submitted to the correct place. Thank you!

[yijiew]

@mcdopsa How did you specify the dependencies in your Unity? Did you copy all the required dlls to the Unity folder or add the body tracking SDK folder to your system path environmental variable?

[mcdopsa]

@mcdopsa How did you specify the dependencies in your Unity? Did you copy all the required dlls to the Unity folder or add the body tracking SDK folder to your system path environmental variable?

Initially, before the breaking update, I was able to use Rumen Filkov's Unity examples out of the box, with dependencies assigned in the Unity root / Plugin folders. This worked well / as expected, and is currently how we are running body tracking in Unity on the half dozen other machines we have operating.

After the update I have tried adding the SDK folders to path environment variable (user and system), several different methods of linking dependencies troubleshooting with the developer directly over screenshare, where we tried various versions of those files in different locations, folders, harddrives (C:/ drive, D:/ drive, Unity root, Plugin folder, etc). Microsoft's newly released SDK contains detailed build instructions on how and where to place dependencies - which have been followed explicitly. Unfortunately each of these tests resulted in the same crash. Throughout this process, the Body Tracking Viewer executable continues to operate without errors.

[yijiew]

I got in touch with the OS team about this issue. It turned out that the problem was in onnxruntime that we depends on to run our body tracking algorithm. The latest windows includes an onnxruntime.dll in the system32 path. However, the onnxruntime.dll we are using in the body tracking SDK is not the same one the windows system is using. Unfortunately they are sharing the same name now. Then it depends on from which folder the executable loads the dll first. Normally, it loads dll first at the same folder as the executable. That is why you see the Body Tracking Viewer works without problem. Then it normally start to search from system32 path. In your case, it found the wrong system onnxruntime.dll and started to load from there. Since it is the wrong one, it failed to load correctly and then crash. (For your reference, you can find the dll search order here )

Thanks for reporting this issue. We will come up with a solution in next release as soon as possible.

[rfilkov]

I can confirm we have tried many (if not all possible) combinations of dll files in both Plugins-folder and the root folder. All without success. Finally, when the same issue popped up in your own Unity sample, it became obvious that the issue is deeper than we initially expected.

According to the DLL search order link above, the current directory is searched before the Windows system directory. This should mean that onnxruntime.dll found in the root folder should be loaded before the onnxruntime.dll located in the system directory. Unless it is already loaded in memory, or the dll search order got changed in the latest 'Windows Insider Build'.

[yijiew]

@rfilkov : Thanks for the information! Since the pdb is not included in this version anyway, it would be very helpful if you could share your crash dump with us. You can save a crash dump by following instruction here: https://docs.microsoft.com/en-us/visualstudio/debugger/using-dump-files?view=vs-2019.

[mcdopsa]

@yijiew Here is the Unity dump file, created through Visual Studio as described in the link you shared. The file is over 1GB so I've made it available through this WeTransfer download. https://we.tl/t-CIfSOIQlgW

Thank you both for all the support!

[yijiew]

Thanks for sending the dump. Just confirmed, this issue is exactly the same as the #932. It crashes at the delay load step because it found the wrong onnxruntime.dll to load:

[rfilkov]

@yijiew Thank you for nailing the bug so quickly! Just out of technical curiosity, what's exactly the issue with onnxruntime.dll loaded from the system folder? If it is a newer release of the library you're using anyway in the BT SDK, shouldn't it be backward compatible? In this case it shouldn't matter, if you're loading the library from the current folder or from the system folder. If your version of onnxruntime is newer or somehow customized, couldn't you load it explicitly from the current folder (or log an error, if it is not there)? What would happen, if onnxruntime was already loaded in memory, when the body tracker gets created?

[yijiew]

Hi @rfilkov The onnxruntime.dll we are using is newer than the one in the system folder. And we use a version with CUDA GPU support while the system one doesn't.

You are right. One way to solve this issue is to try to load the dll explicitly in our SDK. We are exploring this option on our side. Meanwhile, you can imagine that it can be a potential problem for all onnxruntime customers, as onnxruntime itself is a separate shipping SDK. We are also currently actively discussing with their team and the OS team to come up with a solution for this issue.

[qm13]

Tracking in #932