Unable to use transformation matrices outside TouchDesigner

[FilipePires98]

Hello again Darien,

I have been experimenting with TDAzureMerger a bit more and got stuck in trying to use the generated transformation matrices outside of TouchDesigner for perspective alignment.

I have successfully aligned 3 Azure Kinects using TDAzureMerger. I extracted to a .py file each of the transformation matrices available at project1>AzureCalibration>Azure[N]>mtx (where N is either 2 or 3, since Azure1 is the master and does not need any transformation). I have exported point clouds from my Kinects using Open3D and loaded them to CloudCompare (a software similar to MeshLab). However, when I apply a transformation to the subordinate perspectives, the alignment is incorrect and does not match the quality of the one achieved inside TouchDesigner...

Individual Point Clouds inside CloudCompare:

Manual Alignment using CloudCompare Tools:

Automatic Alignment using TDAzureMerger:

Application of the transformation matrix generated by TDAzureMerger on the original subordinate point cloud inside CloudCompare:

NOTE: the clothes are different because the images were taken on different days, but the setup did not change in the mean time.

[FilipePires98]

How can I export the transformation matrices correctly? Is there an additional transformation done inside TouchDesigner that I am missing? Thank you for your time.

[DarienBrito]

Hi Filipe, If you use the matrices generated by the TDAZureMerger that should be it indeed. There's nothing else being applied to the data beyond those transforms, so it sounds like you are doing the right thing.

I'm not sure of what CloudCompare does if you pass those matrices. It should be a simple linear transformation, so they should match one to one. I don't have at the moment kinects to try this out unfortunately. What are the differences number-wise between the matrices generated from CloudCompare and the TDAzureMerger?

[FilipePires98]

Hi Darien,

Here is one matrix produced by TDAzureMerger:

0.9512052466133266 -0.2802859455043801 0.12902855330505805 -0.2507372441724427 0.2675875299617918 0.5410995421084746 -0.7972503993971694 1.309191339061901 0.1536407908864948 0.792875194644602 0.589697747234172 0.48986485618331643 0.0 0.0 0.0 1.0

Here is one matrix produced by aligning 2 point clouds in CloudCompare:

0.957183659077 -0.270673841238 -0.102640494704 0.255080491304 0.236296206713 0.525741994381 0.817165493965 -1.281115293503 -0.167222902179 -0.806431055069 0.567190885544 0.635612607002 0.000000000000 0.000000000000 0.000000000000 1.000000000000

Note that for both these matrices the same azure kinect is used as the master and the same second azure kinect is used as the subordinate. If you want, I can send you example point clouds too.

[Zulex]

@FilipePires98 Did you manage to solve this? I am planning on using Darien's component to do registration inside Unity.

[FilipePires98]

Hi @Zulex, unfortunately I was not able to solve this issue. This is most likely caused by the use of different representations of data for the transformation matrix between CloudCompare and TDAzureMerger, but I was not able to determine which form of representation is used by this tool. Executing registration inside Unity is indeed very desirable, do let me know if you succeed at it. Best of luck!

[DarienBrito]

Hi @Zulex and @FilipePires98

So sorry I missed this message by a long shot! I don't have notifications enabled for Github and cleaning up stuff for the coming year I realized I had these in here.

The used case with CloudCompare is interesting because in my case it works fine. What I did for one production is having frontal calibration done by my tool and calibration of the back camera with CloudCompare using as reference the same one. Once exported, the matrix of CloudCompare used as transform for the point-cloud of the kinect gives correct results. So I do not think that the coordinate spaces or the matrix are dissimilar. There is perhaps something else going on.

I'll be happy to check this on a call or if you send me a patch after the holidays. I'll be back in the studio around 14th January

[FilipePires98]

Hi @DarienBrito , Thank you for your message. Unfortunately, I am no longer working with TD for Azure Kinect calibration. It is possible that the cause of this was related to some parameter(s) inside CloudCompare, idk. Thank you anyways for making yourself available, but by this time it will no longer justify spending more time on this issue. If you wish, you may close it. :)

[DarienBrito]

Alright! Thanks for letting me know Filipe. Happy new year and good luck with it all!