StructureFromKnownPoses resulting in wrong orientation of cameras

[Dave-van-der-Meer]

Issue

When I want to use the plug-in StructureFromKnownPoses I am able to load the poses that I hand over to Meshroom via the cameraInit.sfm file that I generate, but when it imports the poses, the orientation of the cameras is wrong as all the cameras face into the exact same direction (see images below). I generate the camera poses (center and rotation) in a simulation with Gazebo. The orientation is given in Quaternions and I convert them into the 3x3 matrix as necessary for Meshroom.

I also verified if the issue lies not withing the pose data that I provide to Meshroom, but this seems to be correct.

For further verification, I used Meshroom with the same image data to generate the camera poses with the StructureFromMotion node inside the cameras.sfm file and loaded them into my simulation program where I get the camera poses in the first place (see image below). The camera poses from Meshroom (in blue) are in the wrong frame due to the lack of reference which is acceptable, and the cameras are all rotated compared to the direction of movement. This can be compensated with some simple rotations for each camera pose.

After compensating the orientation of the cameras, I have them face in the direction of motion which is what I want. The applied rotations in Euler terms are (0, 0, -pi/2) and (pi/2, 0, 0). When converting the camera poses form my simulation model to Meshroom format, I simple use the exact rotations in reverse order and with opposite signs (hence (-pi/2, 0, 0) and (0, 0, pi/2) which should work).

Question

How can I import my camera pose data so that the orientation is correct and so that not all the cameras show in the exact same direction? Could this be a bug in the software or am I doing something incorrectly?

I add my cameraInit.sfm file here below in case it helps. As the orientation is not correct, Meshroom cannot generate the 3D mesh.

new_cameraInit.zip

I would be glad for any kind of help in this topic.

[natowi]

1 create a small test dataset (~6 images) 2 compare your converted sfm file to the meshroom generated sfm file

/ The cameraInit.sfm does not help here, as it does not contain the poses

[Dave-van-der-Meer]

I did as told and I compared my new_cameraInit.sfm file with the cameras.sfm from StructureFromMotion.

Basically, the poses and the orientations are completely different, but this is also due to the fact that the entire frame, on which each camera pose is based on, is different.

Here is the pose for one camera in my own generated file:

        {
            "pose": {
                "locked": "1", 
                "transform": {
                    "rotation": [
                        "0.00769509303474325", 
                        "5.5674790906956595e-06", 
                        "-0.9999703923177876", 
                        "0.7866020710904122", 
                        "-0.6174306298459542", 
                        "0.00604971770436008", 
                        "-0.6174123154744007", 
                        "-0.7866253347668326", 
                        "-0.004755565529748207"
                    ], 
                    "center": [
                        "0.129007791743", 
                        "-0.155972623641", 
                        "0.498989014823"
                    ]
                }
            }, 
            "poseId": "1811102451"
        }

As compared to the same camera pose generated by Meshroom:

        {
            "poseId": "1811102451",
            "pose": {
                "transform": {
                    "rotation": [
                        "0.73927943849711575",
                        "0.030911758253169386",
                        "0.67268891399895014",
                        "0.0078779923000336458",
                        "0.9984804782235146",
                        "-0.054540552287836538",
                        "-0.67335269291264443",
                        "0.045620146955476101",
                        "0.73791256469791255"
                    ],
                    "center": [
                        "-0.35001690016218062",
                        "0.0018779564669029838",
                        "0.25627730002519017"
                    ]
                },
                "locked": "0"
            }
        }

In addition to that, I see that the intrinsics object generated by Meshrooms is slightly different and has one more entry:

    "intrinsics": [
        {
            "intrinsicId": "3258880732",
            "width": "800",
            "height": "800",
            "serialNumber": "D:\/student_job\/simulation_7a_dataset\/dataset_7a",
            "type": "radial3",
            "initializationMode": "unknown",
            "pxInitialFocalLength": "-1",
            "pxFocalLength": "476.88751019703949",
            "principalPoint": [
                "399.65205714941459",
                "399.35053198751092"
            ],
            "distortionParams": [
                "-0.0021379299300485792",
                "0.001419875983140975",
                "-0.00021830693741415201"
            ],
            "locked": "0"
        }

The newcameraInit.sfm does not contain the "principalPoint" and it has different values for the "distortionParams" and "pxFocalLength"_.

In the end, comparing the poses doesn't make sense unless the two frames are matching which is not the case and I wouldn't know how to achieve this.

Edit: I attach both files for comparison:

cameras.sfm.zip new_cameraInit.sfm.zip

[natowi]

You can take a look at the python script here https://github.com/bastkop/personenscanner/blob/master/convertjson.py

(I just found we had this discussion before https://github.com/alicevision/meshroom/issues/788)

Try to bring your six image dataset sample in the correct sfm format structure (same as cameras.sfm), so it can be excluded as an error source.

Here are a few useful nodes for testing:

• the SfMTransform node with "from single camera" and the view id to align your model to (so the default model is no longer randomly placed in the 3d space)

• the SfMAlignment to align one sfm file to a reference sfm file (and compare the changes)

• the ConvertSFMFormat to convert abc files to sfm

[Dave-van-der-Meer]

Indeed, we had some discussions before. Through them , I was able to load the camera poses inside Meshroom. Now, only the orientation isn't correct.

I can exclude the order of the entries of the .sfm files. I modified them before and it did not make a difference back then. Though, I don't know how Meshroom is exporting the data in this specific order. When I use a JSON export function, I have no clue how to alter the order of the data.

Okay, I looked at the ConvertSfMFormat node, this can be very handy for debugging what is going on. Thanks a lot for this one.

I also looked at the SfMAlignment node, but I don't fully understand how it works, even after reading the documentation. Somehow, I do not get the different frames i.e. the StructureFromMotion and the StructureFromKnownPoses aligned with this node. The aligned frame and the other two frames are all in different frames. Only the size of the aligned frame does match the size of the reference frame.

I finally also had a look at the SfMTransform node with the "from single camera" option. I Understand that the entire frame will be rotated and scaled around the one camera that I provide as input ID, is that correct? How do I apply a specific rotation? I am not sure if this node is what I need to solve my issue that the individual cameras are incorrectly rotated.

Basically, my issue is more or less as to be seen in the following picture:

I also wonder how the elements in the rotation matrix are layed out? I need to convert the orientations that I obtain from ROS into a 3x3 matrix. I read the matrix as follows:

[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Or do I need to read it as follows?

[[1, 4, 7], [2, 5, 8], [3, 6, 9]]

I am not entirely sure with this one. I use the Python module called transforms3d to convert the code from quaternions into 3x3 matrices as follows:

        matrix_3d = tf3d.quaternions.quat2mat([self.qw, self.qx, self.qy, self.qz])
        self.m00 = matrix_3d[0][0]
        self.m01 = matrix_3d[0][1]
        self.m02 = matrix_3d[0][2]
        self.m10 = matrix_3d[1][0]
        self.m11 = matrix_3d[1][1]
        self.m12 = matrix_3d[1][2]
        self.m20 = matrix_3d[2][0]
        self.m21 = matrix_3d[2][1]
        self.m22 = matrix_3d[2][2]
        self.rotation = [str(self.m00), str(self.m01), str(self.m02), str(self.m10), str(self.m11), str(self.m12), str(self.m20), str(self.m21), str(self.m22)]

Is my code correct or am I doing something wrong?

[Dave-van-der-Meer]

Okay, I finally found my mistake:

when applying a rotation to the orientation of the camera as I receive them from Gazebo, I applied the correcting rotation times the current orientation in form of a quaternion multiplication, but I need to inverse the order of both factors. So current orientation times correcting rotation.

[natowi]

Sorry for not coming back on your issue earlier, I have a time consuming project I am working on t the moment. Good to hear that you were able to solve your issue.

If you have some time it would be greatly appreciated if you could summarize and share your workflow/experience/sample code as the question "how to use own camera positions" comes up often and my guide in the wiki is quite basic and experimental.

[Dave-van-der-Meer]

If you have some time it would be greatly appreciated if you could summarize and share your workflow/experience/sample code as the question "how to use own camera positions" comes up often and my guide in the wiki is quite basic and experimental.

At the moment, I am currently quite busy myself, but as soon as I have time, I can explain in more detail how I use Meshroom for my purposes to provide my known camera poses. Currently, I fail at the depth map generation, so I need to solve this issue first.

How should I share my explanations? Shall I open it as an issue or something else? (I am not very familiar with GitHub in this matter.)

[natowi]

You can extend https://github.com/alicevision/meshroom/wiki/Using-known-camera-positions or create a new wiki page.