Add Kinect for Azure Support

[philipbelesky]

Will likely involve splitting or augmenting Core.cs and shifting a number of constants regarding FoV/resolution into some sort of manager.

[philipbelesky]

Ok, some initial challenges/thoughts:

• Currently the library is documented for C/C++. Not a problem per-se, although I haven't done much of interop work in a C# context.
  • A C+ library is completed, and seems to live here and should be able to be cross-referenced vs the C reference here. However, its compiled with .net standard.
  • There is a user developed C# library although compiled for .NET 4.6.1+.
  • Ideally we would stick in C# for accessing, but not sure how tractable the options there are given our .NET version ceiling.
• How to allow users to select which version of the Kinect to look for? Related: how to allow users to configure the depth/color modes?
  • The options component is probably the best bet here as the differences in resolution mean the configurations would require updating anyway.
  • This could be a right-click toggle rather than a parameter, but I'm not sure if that's too obscure? Probably better to have it as something that will persist between definition loads anyway.

[philipbelesky]

Actually, maybe its just easiest to update to .NET 4.6.1? I think that is installed by default on Windows 10 and users can always install it themselves if needed. That would also potentially come with a better JIT compiler and fewer issues with our other dependencies.

[mariuszhermansdorfer]

I think we should upgrade .NET. It's better to do it now before we have shipped the plugin than later when we have a user base counted in millions ;)

Support for Kinect 2 and Azure should definitely persist between sessions. I like the idea of bundling the selection into the options component.

Btw - 4 Kinects Azure DK have just arrived at my desk!

[mariuszhermansdorfer]

OK, merged the PRs you posted. I'm a bit swamped with work now so won't be able to contribute significantly in the coming week, but feel free to ping me with specific questions and issues.

[philipbelesky]

@mariuszhermansdorfer @BarusXXX just a couple of handover notes:

• The official Azure SDK is .NETStandard 2.0, so this library wraps it for 4.6.1
• That same repo has sample implementations that I was using as a reference. This issue was also helpful
• The controller class has been split into a Core and controller for each device type. There may be some replication of properties that haven't been cut down yet. The Azure controller also tries to support different viewer modes (e.g. FoV and resolution)
• CaptureFrame() is called after an init and config function. It pulls the depth picture before converting it to a point cloud.
• I'm not precisely sure where I got up to and may be missing a commit. I think the init/config steps worked and there was a problem in transforming the depth frame. PS @BarusXXX I think that Fody issue you mentioned might require an upgrade to VS 2019?

[BarusXXX]

@philipbelesky @mariuszhermansdorfer , I managed to run it with VS2019. I managed to pinpoint the problem with transforming the depth frame. Currently the depth array is present but all values in it are of 0 value. On each item in the array being, assigned by the sensor that value is going to 1000000. Does the issue that philip pointed to use a transformation that is different than the one you had for the other kinects? link

[BarusXXX]

Thrown at this line; https://github.com/mariuszhermansdorfer/SandWorm/blob/e57a23fce654642efff7a84acbe7aa3187370f75/SandWorm/Sensors/K4AController.cs#L89

System.ArgumentException: 'depthImage must have 0 width in pixels but has 1024. Parameter name: depthImage'

It seems that the transformation requires an array shaped; 0 x 3145728 While we are giving it (1024x1024) @philipbelesky I guess the transform mapping is a known issue? https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1012#issuecomment-571241628

[BarusXXX]

@philipbelesky With depth mode set to DepthMode = DepthMode.WideViewUnbinned the Stride(bytes for each horizontal line) is 2048 and not 6144, which I get is only the Z value. I am not sure that the function in that line is being used correctly. I have started a new branch with my changes for your review.

[BarusXXX]

https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/a227f73b8c78e17db843e7a962a1fabd571c6213/examples/fastpointcloud/main.cpp#L54 Here there is a fast point cloud data fetch example

[BarusXXX]

Some progress at last... In aspect mode I am seeing something and when I exaggerate the z axis I can resolve that there is some depth to the mesh. So ushorts are being mapped to the wrong height? @philipbelesky do you know if there is some formula to covert depth into height? Or is this resolved when the we implement the transform function?

[BarusXXX]

@mariuszhermansdorfer This is the relevant branch we are working on https://github.com/mariuszhermansdorfer/SandWorm/tree/feature/%2361-MAKS-not-k4a

[mariuszhermansdorfer]

do you know if there is some formula to covert depth into height? Or is this resolved when the we implement the transform function?

@BarusXXX, the depth is converted into height in this function: https://github.com/mariuszhermansdorfer/SandWorm/blob/2b53d409f3cc18aa32da5d14bd61c5a1802dead8/SandWorm/Components/BaseKinectComponent.cs#L185

We multiply it by -1 * unitsMultiplier to flip the direction on the Z axis. Kinect measures the distance looking away from the sensor, while we want to have a 2.5D grid growing away from the table.