search

microsoft / psi

Platform for Situated Intelligence
https://github.com/microsoft/psi/wiki
Other
535 stars 94 forks source link

Future of Azure Kinect Component? #116

Open xiangzhi opened 3 years ago

xiangzhi commented 3 years ago

I just found out that the AzureKinect team released a new version of their body tracking API (v1.1). They mention in their changelog that they won't be releasing a NuGet version of the code.

[Breaking Change] No NuGet install.

Since the AzureKinect component relies on the NuGet version of arbt, this likely means we will be stuck at 1.0.1

I was wondering if this is something y'all are aware of and might have a non-NuGet version released in the future? If not, I might take a stab at writing one since my study depends on it.

chitsaw commented 3 years ago

Thanks for letting us know. As of right now, we do not yet have a plan on how to evolve the AzureKinect component. We're following up with the body tracking team to find out more. Are you planning to do this work on Windows or Linux?

xiangzhi commented 3 years ago

No problem! I probably will try to do this on Windows. I have other stuff on my plate right now but I would let y'all know if I make any progress.

xiangzhi commented 3 years ago

I took at stab at the problem and made some progress. The changes turn out to quite minor. Here's a branch with the https://github.com/CMU-TBD/psi/tree/dev/k4abt-1.1

Here's what I changed:

  1. The tracker configuration now lets you set whether to use the lite model and different type of runners (cpu, tensorRT, etc)
  2. Updated the sensor nuGet to 1.4.1 and use path referencing for the body tracking SDK.
  3. Used the environment variable AzureKinectBodyTrackingSDKDir to point to where you install the AzureKinectBodyTrackingSDK. This leads to the copying and hinting of dlls.
  4. Added a try/catch block that solves a bug in the AzureKinectBodyTrackingSDK where an exception is always through at the beginning.

Unfortunately, this is a Windows only fix right now. The Body Tracking SDK team did release the package for Ubuntu 18.04 but I haven't try anything there yet.

Some of changes feel hacky to me and probably have better alternatives. Let me know if you think I should do a pull request on this. I'm also adding this to my own projects and we'll see if anything else breaks 😃

xiangzhi commented 3 years ago

Found the first downside of my approach. Any other project would need to add the following reference to use anything in the Microsoft.Azure.Kinect.BodyTracking namespace

  <ItemGroup>
    <Reference Include="Microsoft.Azure.Kinect.BodyTracking">
      <HintPath>$(AzureKinectBodyTrackingSDKDir)\sdk\netstandard2.0\release\Microsoft.Azure.Kinect.BodyTracking.dll</HintPath>
    </Reference>
  </ItemGroup>