(How to add) support for Survey points?

[lexvandersluijs]

Hi,

Is there a way to provide MVE with known 3D points, and the location of their projections in 2D views?

This is a technique used in e.g. Autodesk ReCap Photo, Acute3D to prevent/correct drift or warping in reconstructions. There they are called 'Survey points', as they have been surveyed by conventional techniques: using a total station, lookup up the points in a floorplan, etc.

In this presentation by Folker Wienertapper, Harald Wuest e.a. (Fraunhofer) they are called Anchor points: http://www.vis.uky.edu/3dimpvt/assets/papers/18th/FolkerWientapper/FolkerWientapper3DIMPVTRecX.pdf, except there they use 3D-3D correspondences. First a 3D point cloud is created, then a few points are identified for which the location is known, then these are moved to the correct location and marked as 'locked' (a feature of SBA), and the bundle adjustment is run again. This should work well, however the correct point in a sparse point cloud is not always easy to find, so 2D-3D correspondences might be easier to specify.

We have to main uses in mind:

• when creating a reconstruction of an indoor environment, it would help in preserving correct scale and preventing warping in case there is a section which has relatively few features, or limited overlap in the photography. We would compute the 3D coordinates of certain points in the floorplan, manually find their 2D location in three or more images, and pass this information to MVE as 'locked' points. The rest of the reconstruction would then have to conform to these points and projections.

• when creating a reconstruction of an object, it would be possible to place a reference marker in view (e.g. using ARToolKit or ARuco markers, or April tags) below it. The 3D positions of the corners of these markers are known exactly, and their 2D locations can be recovered using the tracking library. This would allow us to create 3D reconstructions of objects with an exact scale and a 'logical' orientation, which would be very convenient for one of our use cases.

Looking forward to hearing what you think of this!

BR, Lex

[pmoulon]

May I suggest to take a look to the GCP registration of OpenMVG http://openmvg.readthedocs.io/en/latest/software/ui/SfM/control_points_registration/GCP/

GCP can be used for rigid registration (similarity computation) or non rigid registration as BA constraints. (see here for the implementation https://github.com/openMVG/openMVG/blob/master/src/software/ui/SfM/control_points_registration/mainLayout.cpp#L215)

The rigid registration approach can be implement easily in MVE too.

BTW, as you suggested you can use some marker of known dimension and then resize the reconstruction to those known measurements.

@simonfuhrmann Feel free to remove my post if it is not appropriate (as I spoke about OpenMVG and not MVE code).

[simonfuhrmann]

It's appropriate, and MVE doesn't support this feature. Specifying 2D-3D correspondences is not possible. I would need to look deeper into the issue to implement this, but I don't have the time. And I don't know how important such a feature is for MVE...

[lexvandersluijs]

Hi Simon and Pierre,

Thanks for your answers. Regarding the importance for MVE, a feature like this would help to get the highest-quality reconstruction as possible, especially if they cover a very large area. But maybe the scope of MVE does not extend to very large areas, so in that case it's probably not a high priority.

For the second scenario - automatic scaling and alignment of smaller scans- I think I have found a workaround:

• we use AR markers around the object, with known positions and sizes
• we let MVE reconstruct all points and cameras, also the corners of the markers
• we detect the markers in all photos (if present), recording the 2D coordinates of their corners
• then, to find the corners of the markers in the point cloud we project them into all views, and check which 3D point is closest to a certain marker-corner by checking the distance of the projection to the 2D marker-corner (taking occlusion into account is a thing.. perhaps a bit of user interaction would help to mark this plane, so that we avoid testing points that are not close to this plane)
• then when the corners of the markers have been identified in the point cloud, then the scaling, rotation and translation that is needed to align can be easily computed.
• this transformation is then applied to all points and camera poses

Just thought I'd share this here, in case someone else could use such a feature.

BR, Lex

[simonfuhrmann]

Once you know the transformation parameters, maybe this code helps. It is used to normalize the scene, but with a few tweaks it will do what you want. https://github.com/simonfuhrmann/mve/blob/master/libs/sfm/bundler_incremental.cc#L488