could you please add more details or more annotations to the python file---colmapRansacOutToResidualReconsInput.py

[Yanchuan913]

hello, Palanglois! Thanks for your great work. Your work is exactly what I am looking for. Recently I am trying to reconstruct a scene using your code. Following the step-by-step instructions in the README file, I could not get a good result even though I have adjusted the parameters several times.

• the output result of Line3D++ is good

• run colmapToRANSACInput.py, and I got the line3dpp_output.json file

• run ransac_on_lines, and I got test_input_for_reconstruction.json and test_no_refine_input_for_reconstruction.json

• using the OpenGL viewer you provide, I found the results above is okay!

• run colmapRansacOutToResidualReconsInput.py, and I get the input_lines_from_residuals.json file. The result is bad. the lengths of some lines are much longer than they should be.

• and then I run the command $LINE_RECONS/build/line_based_recons_param -i input_lines_from_residuals.json -o data/ -ext -v

I get a result. But the result is barely recognizable. I could not find the correspondence between the reconstruction result and the original scene. I think the problem lies in the colmapRansacOutToResidualReconsInput.py step.

• then I tried using the result test_input_for_reconstruction.json instead of the input_lines_from_residuals.json to do the reconstruction by running:

line_based_recons_param -i test_input_for_reconstruction.json -o data/ -np 120 -ext -v

and I get a better result.

So I want to know more about the python file(colmapRansacOutToResidualReconsInput.py and colmapToRANSACInput.py) in detail cause I don't know what the python file is used to do exactly. Could you please add more detail to the README file or more annotation to the code?

[palanglois]

Hello @Yanchuan913 !

Thanks a lot for your interest in this work and for the kind word!

Here is the explanation behind the colmapRansacOutToResidualReconsInput.py file.

Consider the following setup as seen from above:

where

• The red triangles represent cameras.
• The blue square represents a pillar in the scene.
• The green line is a 3D line segment triangulated by Line3d++, thanks to the 2 red cameras (among others).
• The red lines on the cameras represent the residuals i.e the 2D line segment on each image that was used to triangulate the green 3D line segment.

Even though each of the 2 residuals do not represent the whole green 3D line segment (because of the occlusion caused by the pillar), Line3d++ is able to merge these observations into a single contiguous 3D line segment.

While this is a nice property of the Line3d++ method, it causes an issue when we compute the visibility term in our method: for the top camera, this creates the following visibility triangle (in yellow):

This is what happens when you directly use the file test_input_for_reconstruction.json. As you can see, the pillar is intersected by the visibility triangle, which may prevent it from being reconstructed (our visibility term encourages emptiness in the visibility triangle).

In order to overcome this issue, I created the colmapRansacOutToResidualReconsInput.py script in order to project the residuals on the 3D line segments and thus split the 3D line for each residual that generated it.

For the top camera, this leads to using the following visibility triangle (in yellow):

I have always had better results doing things that way so far. However, here is my hypothesis regarding what occurs on your scene: the above method generates one 3D line segments per residual. While this addresses the occlusion issue, this may also lead to making the linear program more complicated (maybe some information is used several times when 2 residual projections overlap ?).

Your input would actually be valuable. It would be great if you could share any information regarding occlusion on the scene you're reconstructing.

I hope this helps! Don't hesitate to let me know if you need any further information.

Pierre-Alain

P.S the colmapToRANSACInput.py file is just a format conversion script, it is passive with respect to the data.

[palanglois]

One more thing, could you by any chance try to run the whole pipeline (until the colmapRansacOutToResidualReconsInput.py step included) again with this modified version of Line3d++ and check whether the line segments still look incorrect?

Thanks in advance! Pierre-Alain

[Yanchuan913]

Hello @Yanchuan913 !

Thanks a lot for your interest in this work and for the kind word!

Here is the explanation behind the colmapRansacOutToResidualReconsInput.py file.

Consider the following setup as seen from above:

where

• The red triangles represent cameras.
• The blue square represents a pillar in the scene.
• The green line is a 3D line segment triangulated by Line3d++, thanks to the 2 red cameras (among others).
• The red lines on the cameras represent the residuals i.e the 2D line segment on each image that was used to triangulate the green 3D line segment.

Even though each of the 2 residuals do not represent the whole green 3D line segment (because of the occlusion caused by the pillar), Line3d++ is able to merge these observations into a single contiguous 3D line segment.

While this is a nice property of the Line3d++ method, it causes an issue when we compute the visibility term in our method: for the top camera, this creates the following visibility triangle (in yellow):

This is what happens when you directly use the file test_input_for_reconstruction.json. As you can see, the pillar is intersected by the visibility triangle, which may prevent it from being reconstructed (our visibility term encourages emptiness in the visibility triangle).

In order to overcome this issue, I created the colmapRansacOutToResidualReconsInput.py script in order to project the residuals on the 3D line segments and thus split the 3D line for each residual that generated it.

For the top camera, this leads to using the following visibility triangle (in yellow):

I have always had better results doing things that way so far. However, here is my hypothesis regarding what occurs on your scene: the above method generates one 3D line segments per residual. While this addresses the occlusion issue, this may also lead to making the linear program more complicated (maybe some information is used several times when 2 residual projections overlap ?).

Your input would actually be valuable. It would be great if you could share any information regarding occlusion on the scene you're reconstructing.

I hope this helps! Don't hesitate to let me know if you need any further information.

Pierre-Alain

P.S the colmapToRANSACInput.py file is just a format conversion script, it is passive with respect to the data.

Thanks a lot. Your answer is really easy to understand with those pictures. I am going to use a better line3D++ result(more lines and more accurate) as input and see if it will produce a better result.

[Yanchuan913]

One more thing, could you by any chance try to run the whole pipeline (until the colmapRansacOutToResidualReconsInput.py step included) again with this modified version of Line3d++ and check whether the line segments still look incorrect?

Thanks in advance! Pierre-Alain

OK , I'll tell you once I get the result