Potential Error in main_importKnownPoses.cpp

[mpr-projects]

Hello,

is it possible that there's a mistake in the file main_importKnownPoses.cpp?

I've imported my camera poses with the ImportKnownPoses node. There doesn't seem to be any documentation on the file format but from looking at the code in main_importKnownPoses.cpp it seems that the data should be in a .json file. The file should contain one line per image and each line is a json dictionary with three entries: 'pose', 'forward' and 'up'.

When I put my camera poses into that format (and after accounting for Meshroom's rotated coordinate system) then they import fine and I can see the camera poses in the 3D-viewer by double clicking on the node. But the cameras point in the wrong direction. If I flip the direction of the 'forward' vector and then the cameras point in the right direction but the two (red and green) lines indicating the local camera coordinate system point in the wrong directions (when compared to the result I get from a reconstruction without known poses). I've also tried swapping and flipping the signs of the inputs in as many ways as I could think of and it never seemed right (the reconstruction always failed or gave wrong results) -- of course I may have missed some combination of inputs ...

I'm not familiar with Meshroom's entire codebase so I'm not sure how exactly the coordinates are used/processed internally but I have the impression that line 372 of src/software/convertmain_importKnownPoses.cpp might have to be

rot.row(0) = -up_vec.cross(forward_vec);

instead of

rot.row(0) = up_vec.cross(forward_vec);

Otherwise we'd get a left-handed coordinate system (but I think Meshroom is using a right-handed one, right?). When I adjust the code with that change and compile and run it then the poses import correctly and the reconstruction works fine.

I hope that was detailed enough, otherwise, I describe it in more visually in a video I've just made about some aspects of using Meshroom (see https://youtu.be/XUKu1apUuVE). The section about using known poses starts at time 19:20, the ImportKnownPoses-related part at 22:20 (feedback is welcome :).

[natowi]

@fabiencastan

@mpr-projects great video!

[mpr-projects]

Thank you!

[servantftechnicolor]

Hi

Look like this application is for some external software's file import.

So the conversion you find in the code are using the conventions of this external software.

[mpr-projects]

Hey,

you know I thought that might be the case. That's why I tried permuting the inputs (swapping axes, flipping the sign of individual axes, flipping the direction of the vectors, ...) with the expectation that one of the permutations would satisfy the required input format. I only started looking at the code when none of them worked.

My first impression was that the code (at least the JSON-related section, lines 300-384) was a work-in-progress because, for example, the variables in lines 340-345 are extracted but never used; lines 311 and 363 introduce and populate a vector records which is also never used. And since that part of the code hasn't really been changed since the file was created (~3 years ago) I thought that it might have been abandoned (... and of course lines 372-374 result in a left-handed coordinate system, regardless of the input convention, which seems strange ...).

But if you're sure that it's being used with the output of some program, would you know which program that is or which convention it's using?

[servantftechnicolor]

No, sorry, i don't know anything about this. Did you consider using sfmTransfer or simply modifying json file by script ?

[mpr-projects]

I hadn't thought of using the SfMTransfer node, that's a good idea! For the time being I'll stick with my adjusted ImportKnownPoses node but I'll look into using the SfMTransfer node when I find the time. (I've already tried modifying the json file when I tried the different permutations of the inputs.)

Do you know who owns the file main_importKnownPoses.cpp so we can get more information? Alternatively, do you want me to close the issue? It's not really been resolved because we don't know how that piece of code is intended to be used ... but on the other hand there are workarounds to achieve the desired result ... It may be good to have some documentation about it though.

[natowi]

I agree, this should be documented.