Format of Calibration Results

[jessekirbs]

Could you please explain the format of the end result calibration from this? I see that there are several different files (extrinsics.json in the main Calibration folder, calibration.json in the 'initial_calibration' folder, and calibration.json in the 'crf_refined_calibration' folder) that are output after running the calibration. I'd be interested to try this method's results within something like TouchDesigner - I've tried using the matrices under 'extrinsics' in the the calibration.json files as transforms, but it doesn't seem to be the correct use. If you could go over how to use the calibration output from this method that would be much appreciated. Thank you!

[vladsterz]

Hello @jessekirbs. The folders _initialcalibration and _crfrefined contain the results for intermediate steps of out calibration method. You should be interested only in the base directory files, where the .ply files are results of the calibration procedure for each viewpoint and extrinscis.json contains the transformations. Transformations are stored as 4x4 row-major transformation matrices. Can you describe how did you use these transformations and show examples of your inputs and the corresponding outputs?

[jessekirbs]

Hey @vladsterz - thanks for the clarification! After transposing the matrix from row-major to column-major within TouchDesigner, the pointclouds aligned much better. I'm seeing an offset for some reason which you can see here:

I experimented with doing capturing and calibration several times with different positions for the Kinects and the offset was the same every time. The calibration looks great within your software but when used in TouchDesigner the offset is present. Here's a pic of the transform setup:

The raw matrices are fed into TouchDesigner and converted into tables, then they're transposed to column-major and applied as transforms to each Kinect.

Do you have any idea why I might be seeing this consistent offset? Thank you for the support!

[vladsterz]

Before we proceed any further into debugging, could you please share with me the results of calibration? To do so, please use a point clouds viewer program ( preferred one is meshlab ) in order to merge the final calibrations (files that are in the same directory as calibration.json with .ply extension ) and please show me the result from some descriptive views.

[jessekirbs]

Here's the Meshlab project with the calibration meshes from the two Kinects I used in the above TouchDesigner test. The calibration software was a being a little buggy so I could only get two of the Kinects aligned - the third one would sometimes align with the box structure, but it would be mirrored so it was not usable. Please let me know if I can provide anything else. Thanks, @vladsterz!

calibration.zip

[vladsterz]

Hello, There are missing files, you should've include the actual point-clouds ( .ply files ).

[jessekirbs]

Sorry about that - updated the file link above to include the .ply files.

I'm trying to debug the calibration data further by applying the extrinsics matrices to virtual cameras within Blender to see if they match positions with my Kinects. It looks like they're in the correct positions and angles but are facing away from center instead of in towards the center. Any idea why that might be?

[vladsterz]

I guess that the issue here is the rotation part which is transposed. I think you should look into the way toy are reading out files. Our files contain the extrinsic calibration as an array of numbers, let's say [e1,e2,e3..., e16]. The matrix should be constructed as	e1	e2	e3	e4
e5	e6	e7	e8
e9	e10	e11	e12
e13	e14	e15	e16

[jessekirbs]

That's the way I've been structuring the matrices. The offset in TouchDesigner may be an issue related to that software or the transpose method so I'll look into that further with the devs over there.

In Blender, the cameras should line up and match the Kinect positions when I apply the matrices, correct? I'm using this add-on (https://github.com/SBCV/Blender-Matrix-World-Addon) which allows loading the matrices and applying them.

Thanks, @vladsterz!

[vladsterz]

Indeed, these transformations, when applied to corresponding pointclouds, should align the pointclouds in the calibrated coordinate system. A quick test would be making the transformation matrix as	e1	e5	e9	e4
e2	e6	e10	e8
e3	e7	e11	e12
e13	e14	e15	e16

i.e. transposing only the rotation part of the matrix.

[jessekirbs]

@vladsterz Sorry, can you expand on what you mean by that quick test please? Make the transform matrix only contain:

e1 | e5 | e9 | e4 ?

And would this be a test within Blender? If you could clarify I'd appreciate it. Thanks, vlad!

[vladsterz]

Eh, I know what confused you, its the bold row but I didnt find a way to get rid of it. What I mean is to transpose the rotation part. A rigid transformation of a body in 3D consists of a rotation part (3x3 matrix) and a translation vector (3x1 vector) and is composed as	r1	r2	r3	v1
r4	r5	r6	v2
r7	r8	r9	v3
0	0	0	1

What I say is, transpose the rotation part (the inner rotation 3x3 matrix ).

[jessekirbs]

Ah, I think I understand. You're suggesting transposing the inner 3x3 rotation matrix into column-major format instead of row-major? I tried this but the results were strange in Blender. I also tried rotating the rotation matrix by 180 and that didn't work either. Please let me know if I'm still misunderstanding your suggestion.

I also realized the coordinate spaces for the Kinect are Y Up and Z Forward, while Blender is Z Up and Y Forward which I assume is causing (or exacerbating) the issue. Is this calibration software using Y Up? I'm going to test in Maya which uses Y Up axis.

EDIT: I just tried using the matrices within Maya and still didn't get the correct result. I also tried applying the calibration matrices to pointclouds recorded from each Kinect with no luck. I tried this with the direct matrix output from your software, transposing the rotation matrix, and switching from column-row to column-major but none of them worked. Have you guys used the calibration matrices from your software in third party 3D software successfully? If so, which software did you use? So far only TouchDesigner has come close to looking correct aside from that strange offset. Thanks for your continued help, @vladsterz !

[vladsterz]

Sadly we haven't used any 3d party 3D software with our calibration as far as I know. But our input is X down, Y left, Z in.

[jessekirbs]

I tried entering the matrix values to match your coordinate space but still not having any luck getting the cameras to look right or the pointclouds to align. Something I thought of - I opened an issue on the VCL Volumetric Github regarding my pointclouds being upside down (https://github.com/VCL3D/VolumetricCapture/issues/34). Is it possible this issue is causing the problems I'm seeing here?

[vladsterz]

Sadly we've never encountered such an issue with our applications. I answered to your other post, so try that solution, but to fully support you in fixing this issue I should look into blender and/or TouchDesigner, which I cannot do for some time.

[jessekirbs]

Thanks, @vladsterz. I'll try flipping the Kinects. I guess I assumed the camera should be on top when filming vertically.

I understand. Thanks for your assistance with this and let me know whenever you get a chance to test it out in Blender/TouchDesigner! Looking forward to new volumetric capture release as well. Take care!

[jessekirbs]

@vladsterz Been looking into this more and I wanted to clarify the format of the extrinsic calibration 4x4 matrix. I'm getting confused by the varying coordinate systems and how that effects the matrix. Is the matrix output from your calibration software in this format:

r1(x) r2(y) r3(z) v1(x) r4(x) r5(y) r6(z) v2(y) r7(x) r8(y) r9(z) v3(z) 0 0 0 1

or is it in this format (due to using 'X down, Y left, Z in' coordinate system that you mentioned):

r1(z) r2(x) r3(y) v1(z) r4(z) r5(x) r6(y) v2(x) r7(z) r8(x) r9(y) v3(y) 0 0 0 1

or a variation of that? Is the coordinate system you use considered ZXY Euler? All the variations are scrambling my brain so I'm trying to understand how to translate between programs and their corresponding coordinate systems. Hope this makes sense. Thanks!