search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.5k stars 619 forks source link

Tracker.Create fails on Unity #1147

Closed RamType0 closed 4 years ago

RamType0 commented 4 years ago

Describe the bug It seems that we could not create tracker in Body Tracking SDK 1.0.1, although that was able at Version 1.0.0.

To Reproduce Steps to reproduce the behavior: 1.Setup sample_unity_bodytracking.

2.Create empty GameObject at new scene,and attach this script.

using UnityEngine;
using Microsoft.Azure.Kinect.Sensor;
using Microsoft.Azure.Kinect.BodyTracking;
using System;

public class CreateTracker : MonoBehaviour
{
    // Start is called before the first frame update
    void Start()
    {
        using (Device device = Device.Open())
        {
            device.StartCameras(new Microsoft.Azure.Kinect.Sensor.DeviceConfiguration()
            {
                CameraFPS = FPS.FPS30,
                ColorResolution = ColorResolution.Off,
                DepthMode = DepthMode.NFOV_Unbinned,
                WiredSyncMode = WiredSyncMode.Standalone,
            });

            var deviceCalibration = device.GetCalibration();
            try
            {
                Tracker.Create(deviceCalibration, new TrackerConfiguration() { ProcessingMode = TrackerProcessingMode.Gpu, SensorOrientation = SensorOrientation.Default });
            }
            catch (Exception e)
            {
                Debug.LogException(e);
                return;
            }
            Debug.Log($"Tracker has been successfully created.");

        }
    }
}

3.Enter Play Mode,then you could see that

Tracker has been successfully created.

4.Exit Unity Editor.

5.To Update BodyTrackingSDK to Version1.0.1,Download raw nupkg. 6.Replace libraries(at Assets/Plugins,or Project's root directory)

of Version 1.0.0 with these of Version 1.0.1.(Upgrading from 1.0.0 to 1.0.1 has no dependency changes.) 7.Enter PlayMode, then you could see exception log. like

AzureKinectBodyTrackingCreateException: result = K4A_RESULT_FAILED Microsoft.Azure.Kinect.BodyTracking.Tracker.Create (Microsoft.Azure.Kinect.Sensor.Calibration calibration, Microsoft.Azure.Kinect.BodyTracking.TrackerConfiguration configuration) (at D:/a/1/s/src/csharp/Microsoft.Azure.Kinect.BodyTracking/Tracker.cs:32) CreateTracker.Start () (at Assets/CreateTracker.cs:24) UnityEngine.Debug:LogException(Exception) CreateTracker:Start() (at Assets/CreateTracker.cs:28)

Expected behavior Creating tracker succeeds like before upgrading to 1.0.1.

Desktop (please complete the following information):

Additional context Maybe this isn't a unity specific problem.

yijiew commented 4 years ago

Thanks for reporting the issue. Could you try to generate the k4abt log file according to the instruction here: https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting

yijiew commented 4 years ago

The onnxruntime.dll should be at the same location of the k4abt.dll. Could you check whether it is the case in your project?

RamType0 commented 4 years ago

Thanks for reporting the issue. Could you try to generate the k4abt log file according to the instruction here: https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting

I'll try it on tomorrow,but maybe this is the first time I build C++ project by myself...

The onnxruntime.dll should be at the same location of the k4abt.dll. Could you check whether it is the case in your project?

onnxruntime.dll is placed at project's root directory,but k4abt.dll is currently placed at Assets/Plugins. But the original sample unity project's readme said that they should be placed at there. And it works correctly on version 1.0.0.

Any way, I'll try to place them at same directory.Which directory should I place both of them? project's root directory? or Asset/Plugins?

yijiew commented 4 years ago

@RamType-0 The version 1.0.1 tried to fix an issue that it might acccidentally load the onnxruntime.dll at the system32 folder (The system32 onnxruntime.dll doesn't have some functionalities we need). In the new design, we require the onnxruntime.dll to be placed at the same folder as the k4abt.dll. Please place both of them at the Assets/Plugins folder and give it a try.

RamType0 commented 4 years ago

@RamType-0 The version 1.0.1 tried to fix an issue that it might acccidentally load the onnxruntime.dll at the system32 folder (The system32 onnxruntime.dll doesn't have some functionalities we need). In the new design, we require the onnxruntime.dll to be placed at the same folder as the k4abt.dll. Please place both of them at the Assets/Plugins folder and give it a try.

I tried to place onnxruntime.dll at Assets/Plugins. But creating tracker failed on both of version with exception

AzureKinectBodyTrackingCreateException: result = K4A_RESULT_FAILED Microsoft.Azure.Kinect.BodyTracking.Tracker.Create (Microsoft.Azure.Kinect.Sensor.Calibration calibration, Microsoft.Azure.Kinect.BodyTracking.TrackerConfiguration configuration) (at D:/a/1/s/src/csharp/Microsoft.Azure.Kinect.BodyTracking/Tracker.cs:32) CreateTracker.Start () (at Assets/CreateTracker.cs:24) UnityEngine.Debug:LogException(Exception) CreateTracker:Start() (at Assets/CreateTracker.cs:28)

yijiew commented 4 years ago

Thanks for giving it a try. Could you record a k4abt log by following this instruction? https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting

RamType0 commented 4 years ago

Thanks for giving it a try. Could you record a k4abt log by following this instruction? https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting

I have command that

K4A_ENABLE_LOG_TO_A_FILE=k4a.log K4A_LOG_LEVEL=w

but no logs appears. Maybe this because unity doesnt use installed k4abt. They which in their Assets/Plugins instead.

yijiew commented 4 years ago

Thanks for giving it a try. The logging system should work as long as you are using the k4a.dll and k4abt.dll. Could you try to modify your system environmental variable to add these two line? (In addition to K4A_ENABLE_LOG_TO_A_FILE and K4A_LOG_LEVEL, you need to use separate environmental varialbe to capture K4ABT logs: K4ABT_ENABLE_LOG_TO_A_FILE and K4ABT_LOG_LEVEL) And also, the log file by default is saved to the executable folder. In your case, if you are in unity running the test scene, you might be able to find it at the folder of unity.exe. Please try to pass in a full file path instead of simply the k4a.log so that it could save to the folder you want. (e.g. use C:\Users\XXX\Desktop\k4abt.log to save it to your desktop)

RamType0 commented 4 years ago

I tried with this kind of settings. But no logs appears...

yijiew commented 4 years ago

Hmm... Strange... Could you try to download the msi package from here: https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download. And run the k4abt_simple_3d_viewer.exe to see whether it works?

RamType0 commented 4 years ago

Hmm... Strange... Could you try to download the msi package from here: https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download. And run the k4abt_simple_3d_viewer.exe to see whether it works?

This kind of log appears. It looks like there is no problem. Body tracking works properly.

New logging session started

[2020-04-08 10:24:00.308] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (278): **** Device Info **** [2020-04-08 10:24:00.308] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (279): K4A SDK version: 1.3.0 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (285): Serial Number: 000375694412 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (289): RGB Sensor Version: 1.6.108 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (292): Depth Sensor Version:1.6.79 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (295): Mic Array Version: 1.6.14 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (298): Sensor Config: 6109.7 [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (301): Build type: Release [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (306): Signature type: MSFT [2020-04-08 10:24:00.333] [critical] [t=25432] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (312): **** [2020-04-08 10:24:00.701] [warning] [t=28792] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dewrapper\dewrapper.c (278): depth_engine_thread(). Depth image processing is too slow at 46ms (this may be transient).

yijiew commented 4 years ago

So does it save these log to a file or you see them on the command prompt after setting K4A_ENABLE_LOG_TO_A_FILE=k4a.log

RamType0 commented 4 years ago

So does it save these log to a file or you see them on the command prompt after setting K4A_ENABLE_LOG_TO_A_FILE=k4a.log

I got them from log files.

joel1st commented 4 years ago

I can confirm that the project instructions are in a broken state as of v1.0.1. After running the MoveLibraryFiles.bat script and running the project it reports K4A_RESULT_FAILED when creating the tracker. To fix this you need to copy/cut the onnxruntime.dll into the Assets/Plugins folder.

qm13 commented 4 years ago

@RamType-0 and @joel1st can you verify that @anantley fix has fixed the issue?

RamType0 commented 4 years ago

@RamType-0 and @joel1st can you verify that @anantley fix has fixed the issue?

It was not perfect. I have need to place

at project's root directory instead of Assets/Plugins.

Because these DLL files are ignored in git repository,and previously placed at project's root directory,if you are upgrading from previous version,it works(which is evil).

BTW,it seems like we need to place onnxruntime.dll both in Assets/Plugins and project's root directory. But onnxruntime.dll at project's root directory is not locked while executing body tracking.

anantley commented 4 years ago

Hi RamType-0, thanks for the feedback.

I also see that onnxruntime.dll is in two places. I can’t get the Tracker to open unless it is in both places. Regarding cublas64_100.dll and cudart64_100.dll I have both of them only in the Assets/Plugins folder and it is working fine. What could be the difference between us? I have shut down the editor and restarted and it still works. Best, Angus

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: RamType-0mailto:notifications@github.com Sent: Tuesday, April 14, 2020 7:01 PM To: microsoft/Azure-Kinect-Sensor-SDKmailto:Azure-Kinect-Sensor-SDK@noreply.github.com Cc: Angus Antleymailto:Angus.Antley@microsoft.com; Mentionmailto:mention@noreply.github.com Subject: Re: [microsoft/Azure-Kinect-Sensor-SDK] Tracker.Create fails on Unity (#1147)

@RamType-0https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRamType-0&data=02%7C01%7CAngus.Antley%40microsoft.com%7Cdae44438c9a14b26c71208d7e0e0e95c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225128902231089&sdata=aqEl6AWCgHlt8%2BbWcP05KzHvVgSxvtG9m9tX4EtASMM%3D&reserved=0 and @joel1sthttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjoel1st&data=02%7C01%7CAngus.Antley%40microsoft.com%7Cdae44438c9a14b26c71208d7e0e0e95c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225128902241088&sdata=LMicIaVncDzwi6hGUOfrbNOJFLx1olVaskVvtU3dhek%3D&reserved=0 can you verify that @anantleyhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fanantley&data=02%7C01%7CAngus.Antley%40microsoft.com%7Cdae44438c9a14b26c71208d7e0e0e95c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225128902241088&sdata=FRjM6jDcMp%2FlHwxnDil8fdseWvT2EjwjhL%2FhqQvHozI%3D&reserved=0 fix has fixed the issue?

It was not perfect. I have need to place

at project's root directory.

Because these DLL files are ignored in git repository,and previously placed at project's root directory,if you are upgrading from previous version,it works(which is evil).

BTW,it seems like we need to place onnxruntime.dll both in Assets/Plugins and project's root directory. But onnxruntime.dll at project's root directory is not locked while executing body tracking.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FAzure-Kinect-Sensor-SDK%2Fissues%2F1147%23issuecomment-613771986&data=02%7C01%7CAngus.Antley%40microsoft.com%7Cdae44438c9a14b26c71208d7e0e0e95c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225128902251075&sdata=93bRwbzqDWv8taenjzItqgQzMZIbSqD990CpLpEPZtc%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOLH3HLYPJGRU6FRIGCWWL3RMUIPRANCNFSM4LRT7ZFQ&data=02%7C01%7CAngus.Antley%40microsoft.com%7Cdae44438c9a14b26c71208d7e0e0e95c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225128902251075&sdata=J8EN7DVQvg%2BMusatBt%2Bw0K4BOZWiS6f37ytzOM%2FdGug%3D&reserved=0.

RamType0 commented 4 years ago

What could be the difference between us?

What version of Unity are you using? I'm using Unity 2019.3.6f1.

anantley commented 4 years ago

2018.3.0f2 I will try 2019 and see if that is the diff

Angus

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: RamType-0mailto:notifications@github.com Sent: Wednesday, April 15, 2020 9:58 AM To: microsoft/Azure-Kinect-Sensor-SDKmailto:Azure-Kinect-Sensor-SDK@noreply.github.com Cc: Angus Antleymailto:Angus.Antley@microsoft.com; Mentionmailto:mention@noreply.github.com Subject: Re: [microsoft/Azure-Kinect-Sensor-SDK] Tracker.Create fails on Unity (#1147)

What could be the difference between us?

What version of Unity are you using? I'm using Unity 2019.3.6f1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FAzure-Kinect-Sensor-SDK%2Fissues%2F1147%23issuecomment-614158247&data=02%7C01%7CAngus.Antley%40microsoft.com%7C00bef3b2ee974f402ac108d7e15e3a76%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225667134002422&sdata=G2eZ6AoumCBxb8jMkcJ%2F3JSzTS4EGQFAXaRwVWJcUTA%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOLH3HIR3FL4NROMXT5QXNDRMXRTPANCNFSM4LRT7ZFQ&data=02%7C01%7CAngus.Antley%40microsoft.com%7C00bef3b2ee974f402ac108d7e15e3a76%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225667134012413&sdata=ZHvWrsToLIc3dKFdhX5RuSU%2F9123hzicHn92oKooZN0%3D&reserved=0.

anantley commented 4 years ago

Hi @RamType-0 I tried the scene in Unity version 2019.2.6f1 and it worked fine

satoshiinblack commented 4 years ago

@RamType-0 and @joel1st can you verify that @anantley fix has fixed the issue?

It was not perfect. I have need to place

* cublas64_100.dll

* cudart64_100.dll

at project's root directory instead of Assets/Plugins.

Because these DLL files are ignored in git repository,and previously placed at project's root directory,if you are upgrading from previous version,it works(which is evil).

BTW,it seems like we need to place onnxruntime.dll both in Assets/Plugins and project's root directory. But onnxruntime.dll at project's root directory is not locked while executing body tracking.

This solution does work for me. Thanks :)

RamType0 commented 4 years ago

@anantley Does your body tracking (also) work with

are in project's root directory instead of Assets/Plugins?

If it works,it seems that my solution is better. if it doesnt work, I recommend you to place them in both directory for now(at master branch).

anantley commented 4 years ago

@RamType-0 Hi sorry for the delay, I am now able to recreate your issue and I made a pr to add cublas64_100.dll cudart64_100.dll to the project root.

When it is merged can you verify that this works for you?

Thanks Angus

RamType0 commented 4 years ago

@RamType-0

Hi sorry for the delay, I am now able to recreate your issue and I made a pr to add

cublas64_100.dll

cudart64_100.dll

to the project root.

When it is merged can you verify that this works for you?

Thanks

Angus

@anantley

Thanks! I'll try it on when it merged.

Yuki-Sato-I commented 4 years ago

I also have a headache on this issue.

As described above,

are in my Assets/Plugins and root directory.

but not worked;

The error is

AzureKinectBodyTrackingCreateException: result = K4A_RESULT_FAILED Microsoft.Azure.Kinect.BodyTracking.Tracker.Create (Microsoft.Azure.Kinect.Sensor.Calibration calibration, Microsoft.Azure.Kinect.BodyTracking.TrackerConfiguration configuration) (at :0) AzureKinect.GetDataFromKinect () (at Assets/AzureKinect.cs:77) AzureKinect.Update () (at Assets/AzureKinect.cs:121)

My project version is BodyTracking v 1.0.1 AzureKinectSensor v1.3.0 Unity 2018.3.12f1

Windows10

These packages are installed by Unity Nuget (https://github.com/GlitchEnzo/NuGetForUnity)

Please help :)

BasPaap commented 3 years ago

This still doesn't work. In my project root directory I have cublas64_11.dll, cublasLt64_11.dll, cudart64_110.dll, cudnn64_8.dll, and onnxruntime.dll. These are also in the Assets/Plugins folder. When starting the sample_unity_bodytracking sample from the Azure-Kinect-Samples repository, it still throws an exception on Tracker.Create().

Unity version 2019.1.2f1. Body Tracking v1.0.1 AzureKinectSensor v1.4.1 Windows 10

KikeVilla commented 2 years ago

Somebody found some fix for this issue?. I got the same problem.

berkan20 commented 2 years ago

I had the same issue. In addition to that, I was not able to run Kinect Body Tracking Viewer. I have downgraded the NVIDIA driver to 472.12. This solved the problem with Kinect Body Tracking Viewer. Then I have followed the steps in https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1543. I managed to fix the problem with the location of the dlls. I can run Tracker.Create() now.