Inconsistency with the data at the merger of each cluster. How to use CMVS clustering with openMVS-openMVG?

[ManishSahu53]

I know openMVS is not scalable so I am using clustering option to divide large sfm_data.bin into smaller data.

Specifications like the version of the project, operating system, and hardware

i7-7700k, 16 GB Ram, linux 16.04, 6GB 1060 GTX Nvidia Images are Georeferenced taken from UAV with 16 MP camera.

Steps to reproduce the problem

I am trying to make dense point cloud of large number of images. I am using openMVG in sfm step with clustering option. I have divided sfm_data.bin file into number of sub files(each ~200 images). I could successfully complete dense point cloud and also make DEM using points2grid tool. Now when I was mosaicing the DEM, I observed that there are two different values of DEM near areas of boundary of individual clusters. see image below . Observe two DEM value with very large difference in height values. How can I maintain integrity of the point cloud and DEM so that large difference in elevation occur at junction. Also is there any option of marking area of interest?

[ManishSahu53]

I found a possible solution of this, do correct me if I am wrong. I found out that CMVS is a program that can split sfm data into number of overlapping clusters. So using CMVS with openMVS might solve this problem. My concern is that how to using CMVS and openMVS together as in the internet I could only find CMVS/PMVS combination.

[cdcseacave]

Hi @ManishSahu53,

Indeed, the key here is to use overlapping clusters. There are two ways of achieving consistent reconstruction with it:

1. improve points2grid to be able to support redundancy at the borders and filter outliers
2. create overlapping clusters based on a spatial clustering (CMVS does not support this as far as I know), so that during merging you keep for each cluster only the points inside the grid/assigned volume

I'd go with the second solution. If you plan to contribute this enhancement to OpenMVS, I can support you with guidance if needed.

[ManishSahu53]

Hey @cdcseacave This overlapping clustering idea seems very good. I think this might solve the problem of inconsistency. Yes I would love to contribute in every possible way. I need you guidance since I am relatively new to this. Do you have anything to start regarding overlapping clusters? I am currently using openMVG computecluster funtion to divide the images. However I found out that that function itself produce overlapping clusters. These are the references of computecluster function:

1. Mauro, Massimo, et al. "An integer linear programming model for view selection on overlapping camera clusters." 2014 2nd International Conference on 3D Vision. Vol. 1. IEEE, 2014.
2. Mauro, Massimo, et al. ["Overlapping camera clustering through dominant sets for scalable 3D reconstruction."] (http://www.vision.ee.ethz.ch/~rhayko/paper/bmvc2013_mauro_view_clustering.pdf) 2013 British Machine Vision Conference.

[cdcseacave]

If u plan to implement 1 from the two options I suggested, you are good to go with any of the above papers/methods, including the OpenMVG one. You will need to modify the merging algorithm to be able to filter out outliers at the borders based on redundancy. However, if you go with 2, you will only need to implement a basic spatial clustering and view selection based on the initial sparse point-cloud. This is strait-forward: compute a coarse surface (mesh) from the sparse point-cloud, cluster it imposing a limit on the number of views seeing each cluster (clustering approach can be anything, even a dumb octree could do it), reconstruct each cluster individually using the views selected for each cluster plus some neighbor views for overlapping, and trim the reconstruction of each cluster (can be both the dense point-cloud or the mesh) using the initial cluster borders, and either simply display all of them together, or if you want, merge the meshes in one single mesh.

[ManishSahu53]

@cdcseacave OK but I didnt understand fully. First step - Sparse point cloud to Coarse Mesh Second step - Divide this Mesh according to max num of images ( eg. mesh generated from 1- 50 images are kept as 1 cluster??) I am confused with this point. Please can you elaborate Third step - Reconstruct this mesh cluster to dense by selecting views + some neighbour Fourth step - clip the dense according to starting cluster boundary.

[cdcseacave]

once you have the coarse mesh, we need to cluster it in sections that are seen my mostly N views; one way to do this is to create an octree for the mesh, assign to each cell what views are seeing the contained mesh (by mesh-image projection), and simply cluster the octree so that any cluster has at most N views (this can be done again using an octree, but on the previous octree of the mesh, by dividing a cell as log as the number of views contained are bigger than N)

pls write me if there are still something unclear

[cdcseacave]

here is a more complex method for clustering the surface and the views: https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Zhang_Joint_Camera_Clustering_ICCV_2015_paper.pdf

[ManishSahu53]

[update] I am currently using grid based clustering given here. https://github.com/openMVG/openMVG/issues/857 It is working ok for me. I could get overlapping clusters too. Steps - OpenMVG SFM -> Clutering using grid based (https://github.com/openMVG/openMVG/issues/857) -> Export individual bin file to openMVS -> Run OpenMVS to get Dense Cloud -> Merge individual Cloud using PCL to get seamless point cloud. I used minimum 4 images view to generate points so as to eliminate outliers. [edit] I think the issue is resolved for me so closing this issue.