search

mariuszhermansdorfer / SandWorm

Augmented Reality Sandbox for Grasshopper & Rhino
MIT License
20 stars 11 forks source link

Transformation #68

Closed BarusXXX closed 3 years ago

BarusXXX commented 3 years ago

We need to think if the transformation from depth camera to color camera is needed, also what is the difference if one gets the depth camera and transforms it to color camera using; transformed_depthimage = transformation.DepthImageToColorCamera(capture)

array instead of the grabbing the Z value after transforming the depth imageto a pointcloud via; transformation.DepthImageToPointCloud(depthImage, CalibrationDeviceType.Depth) //the last argument is default so you can omit

or if you have the transformed_depthimage you can use; transformation.DepthImageToPointCloud(transformed_depthimage, CalibrationDeviceType.Color)

since the output after each of these is a image, you can access the underlying array with; myImage.GetPixels().ToArray()

mariuszhermansdorfer commented 3 years ago

@BarusXXX, we don't need to use the color camera to generate the mesh at all. For this purpose, we can easily just work with the depth sensor.

Color image is needed for two other purposes. @philipbelesky, please chime in with other ideas:

  1. Showing a colorized point cloud, where each depth pixel samples the RGB image and takes its color. Example.
  2. Recognition of RGB tokens placed in the sandbox and subsequent analysis.

The reason a transformation matrix needs to be applied is because of the offset and different FOV of the depth and the RGB cameras. As a result, a depth pixel at the nth location in the depth array can't be directly matched to the nth color pixel in the RGB array. These mathematical formulas are provided in the SDK as highly optimized C++ transformation functions. There are several of these and each has a potential use case.

The reason I chose not to use a point cloud generated by the SDK, but went with only the depth pixels was to avoid the additional jitter on the X & Y axes. Since the X & Y coordinate of each pixel are just a function of the camera's FOV and the distance from the sensor, we can just calculate these and have them fixed. This ensures, that we always have a 2.5D grid to work with. In the point cloud, there were some overlapping pixels, which can't be meshed with our algorithm.

BarusXXX commented 3 years ago

image So we do not need bottom left actually, we need If we need top right (which we already have ?!

mariuszhermansdorfer commented 3 years ago

We only need top right for the geodesign course. We can implement the other part later.

BarusXXX commented 3 years ago

@philipbelesky So we can leave the issue with the Transformation result = K4A_RESULT_FAILED error? Since we are getting top left already with; depthFrameData = depthImage.GetPixels<ushort>().ToArray();

At the moment I am also not using any buffer.... so I guess if the kinect collects more data frames then the ones we querying they are being discarded.

philipbelesky commented 3 years ago

Yea, I think we can ignore the transformation method for now — at least until the cloud is shown to have any serious distortions that the transformation is meant to resolve.

BarusXXX commented 3 years ago

Difficult to check distortion without a fixed setup and some objective tests... tomorrow morning I go test it at uni. I will then install my current branch on all machines.

BarusXXX commented 3 years ago

Philip and I realized that the WFOV is to distorted to use directly, flat surfaces appear convex. NFOV less so but I assume it is still convex. A transformation to color camera is needed to make up for this: image

However I get a device discarded error when I introduce the transformation... (see image above) Device here is the object : sensor.

BarusXXX commented 3 years ago

Might be a relevant issue; https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/754

mariuszhermansdorfer commented 3 years ago

Here is an example from another project, that could be of relevance for us: https://github.com/microsoft/Azure-Kinect-Samples/blob/04b87408ec11dddbf9c01321f63a88ad4f654063/build2019/csharp/2%20-%20TransformDepthToColor/MainWindow.xaml.cs#L68

Also, @philipbelesky do you have an idea why I'm not able to build this branch at all? Pulled the latest source, updated NuGet packages via update-package, have zero errors but the solution refuses to build: image

philipbelesky commented 3 years ago

I think this line change changed the way we refer to packages back to an older format. I'm still able to build after a clean/rebuild but I'll see if I can tidy that up and get bundling going at the same time.

philipbelesky commented 3 years ago

Could you both try out 3755dbe4b1abf45519ef166121e81ebf2c59c407 when you have a sec and see if that builds? I've hopefully cleaned things up.

The ability to merge assemblies is constrained because the k4a/depthengine dlls are non-native. So at present they need to be copied over along with the actual Sandworm.gha to the normal Grasshopper components folder.

Note: can we use develop for shared but potentially unstable code, ala GitFlow?

mariuszhermansdorfer commented 3 years ago

Thanks for looking into it @philipbelesky. Unfortunately this one doesn't build nor does it clean. It must be something on my end since it works for both of you. Maybe it has something to do with the recent update to VS that I installed over the weekend...


Microsoft Visual Studio Community 2019
Version 16.7.1
VisualStudio.16.Release/16.7.1+30406.217
Microsoft .NET Framework
Version 4.8.03761

Installed Version: Community

ASP.NET and Web Tools 2019   16.7.532.28833
ASP.NET and Web Tools 2019

Azure App Service Tools v3.0.0   16.7.532.28833
Azure App Service Tools v3.0.0

C# Tools   3.7.0-6.20375.2+34202cc2f3e869fd70a26d8237f4552cf9e192cf
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

IntelliCode Extension   1.0
IntelliCode Visual Studio Extension Detailed Info

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

NuGet Package Manager   5.7.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension   1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

TypeScript Tools   16.0.20702.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   3.7.0-6.20375.2+34202cc2f3e869fd70a26d8237f4552cf9e192cf
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 10.10.0.0 for F# 4.7   16.7.0-beta.20361.3+3ef6f0b514198c0bfa6c2c09fefe41a740b024d5
Microsoft Visual F# Tools 10.10.0.0 for F# 4.7

Visual Studio Code Debug Adapter Host Package   1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio
philipbelesky commented 3 years ago

Does the output from the build step contain any errors? And maybe try nuking the bin/obj/release folders?

I setup an Action to build the project to hopefully track a 'common' build process that theoretically should be able to be used by everyone.

mariuszhermansdorfer commented 3 years ago

Removed all bin/obj/release folders. Killed the .vs folder too. Here is the build log:

Restoring NuGet packages...
To prevent NuGet from restoring packages during build, open the Visual Studio Options dialog, click on the NuGet Package Manager node and uncheck 'Allow NuGet to download missing packages during build.'
Running restore with 12 concurrent jobs.
Reading project file C:\Users\MRHE\Documents\GitHub\SandWorm\SandWorm\SandWorm.csproj.
Reading project file C:\Users\MRHE\Documents\GitHub\SandWorm\SandwormBenchmarks\SandwormBenchmarks.csproj.
The restore inputs for 'SandWorm' have not changed. No further actions are required to complete the restore.
No-Op restore. The cache will not be updated. Path: C:\Users\MRHE\Documents\GitHub\SandWorm\SandWorm\obj\project.nuget.cache
Committing restore...
Assets file has not changed. Skipping assets file writing. Path: C:\Users\MRHE\Documents\GitHub\SandWorm\SandWorm\obj\project.assets.json
Restored C:\Users\MRHE\Documents\GitHub\SandWorm\SandWorm\SandWorm.csproj (in 1 ms).
The restore inputs for 'SandwormBenchmarks' have not changed. No further actions are required to complete the restore.
No-Op restore. The cache will not be updated. Path: C:\Users\MRHE\Documents\GitHub\SandWorm\SandwormBenchmarks\obj\project.nuget.cache
Committing restore...
Assets file has not changed. Skipping assets file writing. Path: C:\Users\MRHE\Documents\GitHub\SandWorm\SandwormBenchmarks\obj\project.assets.json
Restored C:\Users\MRHE\Documents\GitHub\SandWorm\SandwormBenchmarks\SandwormBenchmarks.csproj (in 4 ms).
All projects are up-to-date for restore.
1>------ Build started: Project: SandWorm, Configuration: Debug Any CPU ------
2>------ Build started: Project: SandwormBenchmarks, Configuration: Debug Any CPU ------
========== Build: 0 succeeded, 2 failed, 0 up-to-date, 0 skipped ==========
philipbelesky commented 3 years ago

Ah, I think only @BarusXXX and I talked about this, but we need to do Release and Debug builds on a specific architecture now - the Kinect for Azure dependency won't let us build with Any CPU. So if you change the dropdown to x64 it should hopefully work?

mariuszhermansdorfer commented 3 years ago

@BarusXXX mentioned this already. Unfortunately it doesn't build at all. No matter which architecture I try to compile for.

All projects are up-to-date for restore.
1>------ Build started: Project: SandWorm, Configuration: Release x64 ------
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

Which version of VS are you on? I'll try reverting to it and rebuilding.

philipbelesky commented 3 years ago

I'm 16.7.1

mariuszhermansdorfer commented 3 years ago

Me too. This rules out the VS version issue. I spent the better part of yesterday troubleshooting this issue and am out of ideas now...

philipbelesky commented 3 years ago

Ok, do you want to proceed with the feature/MAKS... branch and we can figure it out later? I can bring anything useful across to there.

mariuszhermansdorfer commented 3 years ago

Yes please. This would be great. Once we have a merged version of your and Kane's code we can then adopt the development scheme you proposed.

mariuszhermansdorfer commented 3 years ago

I have finally figured it out! @philipbelesky, @BarusXXX, after 2 days of troubleshooting, repairing and finally running a clean installation it still wouldn't build. As usual, the issue was super trivial. A recent update to VS replaced my shortcut which defaulted to running it as admin. Without it, my user account didn't have enough rights to build any solutions.

Now I'm back in development mode :)