Scalable pipeline - Githubissues

cdcseacave commented 5 years ago

Add support for processing scenes of any size using a divide & conquer approach:

[x] break the big SfM reconstruction into smaller chunks (sub-scenes) using some kind of clustering (something like https://github.com/srivathsanmurali/domsetLibrary) with overlapping cameras
[x] determine exact boundaries between the sub-scenes using SVM (https://docs.opencv.org/3.4/d1/d73/tutorial_introduction_to_svm.html)
[x] reconstruct each one of them individually using OpenMVS, all the way except texturing
[x] cut each sub-scene mesh to the boundaries computed by SVM
[ ] merge them back (at this point the sub-scenes are meshes) using a mesh zipping algorithm (ex. art 5.3 in http://imagine.enpc.fr/publications/papers/PhD011.pdf)
[ ] apply the texture on each mesh (last module on OpenMVS)

pmoulon commented 5 years ago

Seems like a good plan.

You can also look to this paper: Scalable Surface Reconstruction from Point Clouds with Extreme Scale and Density Diversity, C. Mostegel, R. Prettenthaler, F. Fraundorfer and H. Bischof. CVPR 2017.

If you want to play with domsetLibrary, we already have it implemented for OpenMVG scenes (see here).

rjanvier commented 5 years ago

@cdcseacave thanks for this roadmap. I'm a bit unsure about the SVM thing. for now I assume we use the sparse point clouds from SfM and each point is labeled according to its cluster number, am I right?

cdcseacave commented 5 years ago

@pmoulon thx, I didn't read this exact paper yet (but an other one with a similar idea) @rjanvier the clustering is more for evenly partition the images in sub-sets, the sparse point cloud is only used to compute the best partition for the cameras; I mean we do not label the points, but only the images (plus add some overlap - some images will be part of two or more sub-sets); even if we would label the points, that will still not help us precisely know the boundaries in space between the sub-scenes; in other words, later when we have to merge back the sub-sets, to know exactly which triangles we should take from each sub-set; for that we need to partition the space also, not only the cameras and SVM can be used to estimate those boundaries (ideally a triangle should be piked in the end from the sub-set that best sees it - most cameras seeing it); so SVM can be used at the end when we have all the meshes computed, but that will be insanely slow, and I think it is enough to use only the positions of the cameras and few points from the sparse point-cloud seen by each camera

cdcseacave commented 5 years ago

pls note that I wrote SVM as the second step only to emphasize that it uses the sparse input and not the end meshes, but the computed boundaries are not used until merge step; in other words the sub-set meshes at that point will overlap considerably, and the SVM computed boundaries will be used to precisely cut the overlaps: no two meshes will overlap after that, but will be one close to the next (at the distance of one triangle); so the merging simply needs to unite the boundary vertices and form new triangles that perfectly fill the little space between the meshes

this is just one possibility, an other way is to completely skip computing exact boundaries and somehow during merging determine the best place where the meshes should be cut and stitched together; but I do not see an easy solution to that

rjanvier commented 5 years ago

@cdcseacave thanks for clarifying. so my assumption was right, step two is: determine exact boundaries between the sub-scenes using SVM (https://docs.opencv.org/3.4/d1/d73/tutorial_introduction_to_svm.html) on the sparse point cloud from SfM. Input label for SVM is computed in step one: each camera and track is labeled according to their cluster. Due to the overlapping nature of step one, a track and one camera could be labeled in multiple cluster at time

@pmoulon thx, I didn't read this exact paper yet (but an other one with a similar idea)

@cdcseacave I previously only know Vu's Ph.D and Mostgel's paper on this topic. @cdcseacave is it possible to share your other reference here?

cdcseacave commented 5 years ago

The camera overlap is added so that the sub-set reconstructions have overlap meshes. However for the SVM boundary estimation non-overlaping sub-sets must be used.

My mistake, I read the paper today and it turns out that this is the paper I had in mind, just that I read it long time ago and I forgot the name. The proposed solution is interesting, but has some cons:

does not integrate well with the rest of MVS stages:
- they do not say how the dense point cloud is obtained, but I imagine they must use some clustering for that (so duplication of the work)
- the point cloud is directly converted to one full mesh, so the mesh refinement and mesh texturing stages do not benefit from the clustering
the mesh zipping proposed is close to the proposed approach in Vu's paper, but seems more complex with no significant benefit (the mesh overlaps after refinement should be so similar that a simple zipping should be ok)
it is for sure much slower than the proposed solution, as they use a lot of overlap

rjanvier commented 5 years ago

Ok that clearly make sense for SVM now. I'll try to work on that and push something.

Just few comments regarding Mostegel's paper, I think it's more general in some aspects. Only speaking of meshing (and not refinement neither texturing) I think it could be beneficial to be "clustering-agnostic" on some situations because it could be easier to integrate data from different sources (for example TLS). But I agree with you it must be slower.

rjanvier commented 5 years ago

for now, after few experiments I would say that the SVM with Linear kernel is somehow too limited (or maybe I used it incorectly) and other alternatives (RBF kernel) are too potentially slow. To be sure, which SVM kernel did you plan to use?

cdcseacave commented 5 years ago

from my experience SVM::C_SVC plus SVM::RBF with auto-training should work fine and fast enough; the key is not having too many points to cluster; I suggest to use 2 points per image:

1 point for the camera position
1 point for the average scene location as seen from that image (basically the average of all the sparse points that image is seeing)

You can also try using 4 points instead of 1 for the scene, that is split the image in 4 equal regions, and take the average in each region.

what do you think?

rjanvier commented 5 years ago

Ok thank you and nice to see it's not a linear kernel! My mind couldn't imagine that such "limited" amount of points would be enough ;-). Maybe i'll try ASAP the option with 4 because it seems a good approx of the frustum of the camera.

rjanvier commented 5 years ago

So far I have tuned the current OpenMVG clustering binary in order to compute the SVM classification and export it along with clusters. I will integrate this as a proper OpenMVS binary once I will be done with the other parts. if you want to check the svm parameters, the code is here. After that, I run the "classic" pipeline: I exported each cluster with the openMVG2openMVS binary, run DensifypointCloud and ReconstructMesh. After I reuse the SVM parameters with a Q&D binary I made with OpenMVS (SvmCutMesh). it cuts each cluster mesh by calling the svm prediction function on points of the mesh to see if they share the same label than the one of the current cluster. for now I just kill every face that have any of its vertices that belongs to another cluster. Results are encouraging but it tends to produce "holes" where sparse point cloud was very sparse. Sometime It's not "incorrect" because it occurs on truly weakly supported area. But sometimes it occurs on something sparse on the sparse point cloud that get dense on de dense one. Maybe we should generate our few samples dedicated to SVM from the dense point cloud. Do you think it's a good idea?

Here are an example on a simple building but a complicated survey structure because of different level of zoom/Details.

sparse cloud

sparse

mesh (from from resolution-level:3)

mesh

I used a grid of 16 cell on each image and I pruned cells where there less than 5 visible tracks (it was better than with "only" 4 cells per images)

cdcseacave commented 5 years ago

Impressive job, thx @rjanvier!!!

Right now I do not have time to check closely, I'll try tomorrow. Now I just want to say to also put in place a good way to visualize all the data for debugging easily. For example it would be helpful if you can render each piece in a different color.

On an other hand, the clustering should be done inside OpenMVS, it is not optional, cause as it is not it is not 100% correct. You need to export first the entire scene to MVS, after that call the split module that store the subsets in scene_0000.mvs naming for ex, with the mention to also fill the ID with the global index. Next you call densification module on the original scene.mvs and only compute and store the depth maps (exit before fusion). And only next call densification for each cluster that will simply load the already stored depth-maps and just do the fusion (this is where the global ID is needed).

In this way not only you save a lot of time for not recomputing depth-maps for overlap images, but also ensure the exact same depth-maps are used for each cluster.

rjanvier commented 5 years ago

Well thanks, it was not so hard so far because of your mentoring and also because the clustering was already in a good shape in OpenMVG. So first thanks to the community ;)

Regarding the color issue, you are 100% right. I think I will add an optional color field to vertices of your MVS::Mesh class in order to be able to visualize the whole classification of a given cluster mesh (the faces that could be labeled as a part of the current cluster AND the face that could be labeled as a part of other clusters) this way we could visualize the putative overlap. Do you have a way to generate (seeded) ramdom colors programmatically in openMVS or should I add one?

Thanks for pointing me the correct ID field to use. BTW, do you think its possible to add an optional clusterID to the current interface?

cdcseacave commented 5 years ago

I didn't have time today either to look over your code, I am very busy these days at CVPR

Do not remember if there is random collor generation, pls check TPixel or TPoint3 class for a Random variable. You can also adapt the Viewer inside OpenMVS to display the clusters.

a clusterID could work, but I prefer not to add it, as it works just the same if you code the cluster ID in the file name as: scene_0001.mvs and that does not need to change the interface which is already used in many-many places

KeoChi commented 5 years ago

@rjanvier Hello! Can you provide a python script to execute a series of programs? This makes it clearer to understand the entire process and related parameters. In addition, I want to merge each point cloud after DensifyPointCloud. Can you guide me? Thank you.

rjanvier commented 5 years ago

@KeoChi thanks for your interest, unfortunately it is still very experimental and I do not have any python script with the full pipeline (for now, I use simple bash script for my tests in development). if you know a bit OpenMVG and OpenMVS it should be easy for you to figure out how it works (see this comment https://github.com/cdcseacave/openMVS/issues/438#issuecomment-502288671). but I suggest you should not use this yet because I just add a bunch of glue code to already existing things in order to test my understanding of theoretical aspects quickly so it's even far from an alpha version.

In the current pipeline there is no way (yet) to merge files. If you want to simply merge ply (concat) file I suggest you use cloudcompare see http://www.cloudcompare.org/doc/wiki/index.php?title=Merge . Good luck.

KeoChi commented 5 years ago

I have done some related attempts these days. I still have a lot of questions. 1.I used the openMVG main_ComputeClusters to break the big SfM reconstruction into smaller chunks. Need to customize the size of clusterSizeLowerBound and clusterSizeUpperBound. Is there any good way to set this value according to the performance of the computer(like the memory size)? 2.I used openMVS DensifyPointCloud to reconstruct point cloud for every small chunk. And then each point cloud have overlapping area and a lot of repeating points, which causes the total number of point clouds after the merger of each block point cloud is very large. I think this is unreasonable, but I have not thought of a better way at the moment. 3.I used same dataset(180 images) and computer(8G memory, 4 core i5, no gpu) to test the pix4d. I find some interesting things from the pix4d log file.

[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Writing D:\cpk/test1/temp/dense/test1_gidx.tmp
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Clustering cameras 
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Reading D:\cpk/test1/temp/dense/test1_gidx.tmp
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Writing D:\cpk/test1/temp/dense/test1_cidx_part_1.tmp
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Using 1 cluster
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Reading D:\cpk/test1/temp/dense/test1_cidx_part_1.tmp
[2019.06.19 17:25:12][ 27%RAM][  0%CPU][Info]: Initializing 185 cameras 
[2019.06.19 17:27:25][ 55%RAM][  0%CPU][Info]: Initializing 248938 points 
[2019.06.19 17:27:44][ 55%RAM][  0%CPU][Info]: Expanding from 241269 points 
[2019.06.19 17:29:06][ 56%RAM][  0%CPU][Info]: Generating 1000000 points (3.687%)
[2019.06.19 17:30:26][ 57%RAM][  0%CPU][Info]: Generating 2000000 points (6.925%)
[2019.06.19 17:31:44][ 57%RAM][  0%CPU][Info]: Generating 3000000 points (9.842%)
[2019.06.19 17:33:02][ 58%RAM][  0%CPU][Info]: Generating 4000000 points (12.54%)
[2019.06.19 17:34:19][ 58%RAM][  0%CPU][Info]: Generating 5000000 points (15.123%)
[2019.06.19 17:35:38][ 58%RAM][  0%CPU][Info]: Generating 6000000 points (17.594%)
[2019.06.19 17:37:00][ 59%RAM][  0%CPU][Info]: Generating 7000000 points (19.976%)
[2019.06.19 17:38:22][ 60%RAM][  0%CPU][Info]: Generating 8000000 points (22.274%)
[2019.06.19 17:39:46][ 60%RAM][  0%CPU][Info]: Generating 9000000 points (24.522%)
[2019.06.19 17:41:11][ 60%RAM][  0%CPU][Info]: Generating 10000000 points (26.732%)
[2019.06.19 17:42:36][ 61%RAM][  0%CPU][Info]: Generating 11000000 points (28.869%)
[2019.06.19 17:44:02][ 62%RAM][  0%CPU][Info]: Generating 12000000 points (30.969%)
[2019.06.19 17:45:28][ 62%RAM][  0%CPU][Info]: Generating 13000000 points (33.014%)
[2019.06.19 17:46:49][ 63%RAM][  0%CPU][Info]: Generating 14000000 points (35.012%)
[2019.06.19 17:48:11][ 63%RAM][  0%CPU][Info]: Generating 15000000 points (37.003%)
[2019.06.19 17:49:39][ 64%RAM][  0%CPU][Info]: Generating 16000000 points (38.953%)
[2019.06.19 17:51:00][ 64%RAM][  0%CPU][Info]: Generating 17000000 points (40.842%)
[2019.06.19 17:52:15][ 65%RAM][  0%CPU][Info]: Generating 18000000 points (42.681%)
[2019.06.19 17:53:32][ 65%RAM][  0%CPU][Info]: Generating 19000000 points (44.553%)
[2019.06.19 17:54:56][ 66%RAM][  0%CPU][Info]: Generating 20000000 points (46.565%)
[2019.06.19 17:56:28][ 66%RAM][  0%CPU][Info]: Generating 21000000 points (48.501%)
[2019.06.19 17:57:52][ 67%RAM][  0%CPU][Info]: Generating 22000000 points (50.367%)
[2019.06.19 17:59:09][ 67%RAM][  0%CPU][Info]: Generating 23000000 points (52.425%)
[2019.06.19 18:00:28][ 68%RAM][  0%CPU][Info]: Generating 24000000 points (55.356%)
[2019.06.19 18:01:54][ 68%RAM][  0%CPU][Info]: Generating 25000000 points (59.683%)
[2019.06.19 18:03:14][ 69%RAM][  0%CPU][Info]: Generating 26000000 points (62.377%)
[2019.06.19 18:04:35][ 69%RAM][  0%CPU][Info]: Generating 27000000 points (65.752%)
[2019.06.19 18:06:02][ 70%RAM][  0%CPU][Info]: Generating 28000000 points (69.934%)
[2019.06.19 18:07:11][ 70%RAM][  0%CPU][Info]: Generating 29000000 points (73.611%)
[2019.06.19 18:08:21][ 71%RAM][  0%CPU][Info]: Generating 30000000 points (79.537%)
[2019.06.19 18:09:39][ 71%RAM][  0%CPU][Info]: Generating 31000000 points (87.696%)
[2019.06.19 18:10:51][ 72%RAM][  0%CPU][Info]: Generating 32000000 points (93.909%)
[2019.06.19 18:11:42][ 68%RAM][  0%CPU][Info]: Optimizing 32740564 points [1/3] 
[2019.06.19 18:12:36][ 67%RAM][  0%CPU][Info]: Optimizing 32207380 points [2/3] 
[2019.06.19 18:12:47][ 67%RAM][  0%CPU][Info]: Optimizing 31918713 points [3/3] 
[2019.06.19 18:13:12][ 67%RAM][  0%CPU][Info]: Finalizing 31897584 points 
[2019.06.19 18:13:13][ 67%RAM][  0%CPU][Info]: Writing cluster Vis file <D:\cpk/test1/temp/dense/test1_data_part_1.tmp>

Pix4d only use 1 cluster, this means calculating point clouds from all images. And there is no memory overflow. Only one hour! I don't understanding how it compute point cloud by expanding from points of SFM(Expanding from 241269 points).
In addition, I use openMVS to compute point cloud for 180 images and I encountered a memory error when FuseDepthMaps().

3.I used same dataset(180 images) and computer(8G memory, 4 core i5, 2G gpu) to test the pix4d. The pix4d can use the GPU to compute point clouds. The pix4d log file:

[2019.06.19 15:10:28][ 73%RAM][100%CPU][Info]: Clustering cameras 
[2019.06.19 15:10:28][ 73%RAM][100%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_gidx.tmp
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Writing F:\xmission_dataset/output/temp/dense/output_cidx_part_1.tmp
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Writing F:\xmission_dataset/output/temp/dense/output_cidx_part_2.tmp
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Writing F:\xmission_dataset/output/temp/dense/output_cidx_part_3.tmp
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Using 3 cluster
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_cidx_part_1.tmp
[2019.06.19 15:10:30][ 72%RAM][ 52%CPU][Info]: Cluster 1 of 3 : initializing 78 cameras 
[2019.06.19 15:11:27][ 82%RAM][ 74%CPU][Info]: Cluster 1 of 3 : initializing 137297 points 
[2019.06.19 15:11:42][ 82%RAM][ 99%CPU][Info]: Cluster 1 of 3 : expanding from 96876 points 
[2019.06.19 15:14:29][ 86%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 1000000 points (8.049%)
[2019.06.19 15:16:55][ 84%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 2000000 points (14.405%)
[2019.06.19 15:19:14][ 85%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 3000000 points (20.156%)
[2019.06.19 15:21:37][ 62%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 4000000 points (25.528%)
[2019.06.19 15:24:03][ 62%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 5000000 points (30.563%)
[2019.06.19 15:26:28][ 61%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 6000000 points (35.341%)
[2019.06.19 15:28:47][ 62%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 7000000 points (39.908%)
[2019.06.19 15:31:18][ 65%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 8000000 points (44.273%)
[2019.06.19 15:33:44][ 65%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 9000000 points (48.57%)
[2019.06.19 15:35:51][ 65%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 10000000 points (53.6%)
[2019.06.19 15:38:06][ 65%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 11000000 points (61.399%)
[2019.06.19 15:40:23][ 69%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 12000000 points (70.561%)
[2019.06.19 15:42:30][ 70%RAM][100%CPU][Info]: Cluster 1 of 3 : generating 13000000 points (85.368%)
[2019.06.19 15:44:19][ 70%RAM][100%CPU][Info]: Cluster 1 of 3 : optimizing 13955597 points [1/3] 
[2019.06.19 15:44:41][ 68%RAM][100%CPU][Info]: Cluster 1 of 3 : optimizing 13725188 points [2/3] 
[2019.06.19 15:44:49][ 68%RAM][ 99%CPU][Info]: Cluster 1 of 3 : optimizing 13602169 points [3/3] 
[2019.06.19 15:45:06][ 68%RAM][ 99%CPU][Info]: Cluster 1 of 3 : finalizing 13592610 points 
[2019.06.19 15:45:07][ 68%RAM][ 99%CPU][Info]: Writing F:\xmission_dataset/output/temp/dense/output_tex_part_1.tmp
[2019.06.19 15:45:18][ 70%RAM][ 90%CPU][Info]: Writing cluster Vis file <F:\xmission_dataset/output/temp/dense/output_data_part_1.tmp>
[2019.06.19 15:45:20][ 59%RAM][ 27%CPU][Info]: Cluster 1 of 3 : coloring 13592610 points 
[2019.06.19 15:46:03][ 57%RAM][ 64%CPU][Info]: Writing P4B <F:\xmission_dataset/output/temp/dense/output_raw_part_1.tmp> for 13592610 points
[2019.06.19 15:46:09][ 54%RAM][ 40%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_cidx_part_2.tmp
[2019.06.19 15:46:09][ 54%RAM][ 40%CPU][Info]: Cluster 2 of 3 : initializing 78 cameras 
[2019.06.19 15:47:09][ 63%RAM][ 63%CPU][Info]: Cluster 2 of 3 : initializing 124416 points 
[2019.06.19 15:47:24][ 63%RAM][ 98%CPU][Info]: Cluster 2 of 3 : reusing points from cluster 1 
[2019.06.19 15:47:24][ 63%RAM][ 98%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_raw_part_1.tmp
[2019.06.19 15:47:24][ 63%RAM][ 98%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_tex_part_1.tmp
[2019.06.19 15:48:23][ 66%RAM][100%CPU][Info]: Cluster 2 of 3 : expanding from 6432464 points 
[2019.06.19 15:51:15][ 67%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 1000000 points (31.757%)
[2019.06.19 15:53:56][ 68%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 2000000 points (37.569%)
[2019.06.19 15:56:38][ 68%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 3000000 points (43.293%)
[2019.06.19 15:59:27][ 69%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 4000000 points (49.637%)
[2019.06.19 16:01:49][ 69%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 5000000 points (56.401%)
[2019.06.19 16:04:19][ 70%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 6000000 points (64.517%)
[2019.06.19 16:06:32][ 70%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 7000000 points (73.683%)
[2019.06.19 16:08:38][ 71%RAM][100%CPU][Info]: Cluster 2 of 3 : generating 8000000 points (83.383%)
[2019.06.19 16:10:35][ 70%RAM][100%CPU][Info]: Cluster 2 of 3 : optimizing 8898551 points [1/3] 
[2019.06.19 16:10:50][ 68%RAM][100%CPU][Info]: Cluster 2 of 3 : optimizing 8705293 points [2/3] 
[2019.06.19 16:10:57][ 68%RAM][ 97%CPU][Info]: Cluster 2 of 3 : optimizing 8643112 points [3/3] 
[2019.06.19 16:11:08][ 68%RAM][ 98%CPU][Info]: Cluster 2 of 3 : finalizing 8637440 points 
[2019.06.19 16:11:09][ 65%RAM][ 98%CPU][Info]: Writing F:\xmission_dataset/output/temp/dense/output_tex_part_2.tmp
[2019.06.19 16:11:15][ 68%RAM][ 84%CPU][Info]: Writing cluster Vis file <F:\xmission_dataset/output/temp/dense/output_data_part_2.tmp>
[2019.06.19 16:11:16][ 58%RAM][ 84%CPU][Info]: Cluster 2 of 3 : coloring 8637440 points 
[2019.06.19 16:11:51][ 54%RAM][ 58%CPU][Info]: Writing P4B <F:\xmission_dataset/output/temp/dense/output_raw_part_2.tmp> for 8637440 points
[2019.06.19 16:11:53][ 52%RAM][ 46%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_cidx_part_3.tmp
[2019.06.19 16:11:53][ 52%RAM][ 46%CPU][Info]: Cluster 3 of 3 : initializing 78 cameras 
[2019.06.19 16:12:47][ 60%RAM][ 72%CPU][Info]: Cluster 3 of 3 : initializing 114752 points 
[2019.06.19 16:12:57][ 60%RAM][ 97%CPU][Info]: Cluster 3 of 3 : reusing points from cluster 1 
[2019.06.19 16:12:57][ 60%RAM][ 97%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_raw_part_1.tmp
[2019.06.19 16:12:57][ 60%RAM][ 97%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_tex_part_1.tmp
[2019.06.19 16:13:51][ 64%RAM][ 94%CPU][Info]: Cluster 3 of 3 : reusing points from cluster 2 
[2019.06.19 16:13:51][ 64%RAM][ 94%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_raw_part_2.tmp
[2019.06.19 16:13:51][ 64%RAM][ 94%CPU][Info]: Reading F:\xmission_dataset/output/temp/dense/output_tex_part_2.tmp
[2019.06.19 16:14:06][ 64%RAM][100%CPU][Info]: Cluster 3 of 3 : expanding from 6372465 points 
[2019.06.19 16:16:19][ 65%RAM][100%CPU][Info]: Cluster 3 of 3 : generating 1000000 points (31.496%)

Since the memory of GPU is only 2G, pix4d use 3 cluster. When it computes point cloud of cluster 2, it can reuse points from cluster 1. Does anyone understand this method of pix4d? @cdcseacave @pmoulon @rjanvier Masters, Can you express your opinions? Can you give some guidance? Thank you!

pmoulon commented 5 years ago

@KeoChi We will not reverse engineering Pix4D. Note that you can play with the -resolution-level option in order to process the dataset with less resolution and handle quite a large dataset on a decent PC (We don't know what Pix4D is doing and we will not really look to it since we want to do our own version).

rjanvier commented 5 years ago

Hi.

the good number is function of your RAM and the resolution of your images (And of course the resolution level used in DensifyPointCloud.) I don't have the correct formula right now but it should be easy to figure it out from the current code and it will arrive "when it's done".
Once again please note you use an experimental pipeline. What we expect to optimize right now is not what you expect to get optimized on your side and right now, the pipeline is still not what @cdcseacave has planned initially (see https://github.com/cdcseacave/openMVS/issues/438#issuecomment-502310873).
I don't know what is used by Pix4D internally (I just remember they used Poisson Reconstruction for their Surface Reconstruction step last time I tried it in the end of 2017) maybe you could read about work from E. Tola (https://infoscience.epfl.ch/record/138785/files/tola_daisy_pami_1.pdf). But I'm not sure it was ever used in Pix4D. "Expanding from point": most MVS algorithm use sparse point clouds from the SfM part as seeds to initialize DepthMap/Densification step. The computation time will depends on the resolution level you run the densification. By default openMVS do the depth map computation at half resolution like Pix4d... but Pix4d "average" things every 4 pixels (See Point Density parameters), it's maybe why it's "faster" and less dense. But here you can see openMVS is outperforming Pix4d in terms of quality (even if benchmarks are not 100% reliable). Without knowing parameters used by your test and methods used by pix4D we are comparing oranges and apples here. Also in terms of manpower and budget your are comparing two incomparable things.

As a final note, it's the fusion step that could take down your computer, not the depth map computation itself. Moreover, the strategy is planned to tackle not only the "non scalability" of depth map computation but also the "non scalability " of the surface reconstruction algorithm of OpenMVS. For example for now you can use Poisson Recon at the end of the DensifyPointCloud pipeline which is 100% out of core and can virtually reconstruct input of any size, but the delaunay + graph cut approach of OpenMVS need some love to achieve such level of scalability. Both methods has their strengths and their weakness and your choice could depend on some specifics use cases.

Thanks for your feedback but please do not hijack this thread.

KeoChi commented 5 years ago

@pmoulon @rjanvier Thank you very much for your reply. I used to ask related questions here#444, and then it was closed. So I am asking some questions here. I am very sorry to cause some interruptions. @rjanvier Thank you for your good suggestion.

rjanvier commented 5 years ago

@cdcseacave: I'm eventually in the process on integrating domset library into OpenMVS. Regarding the ID pointed in this comment (https://github.com/cdcseacave/openMVS/issues/438#issuecomment-502288671) it's available in Interface::Image but not in MVS::Image so it can't be kept while using Scene::Load/Scene::Write. should I add it to the MVS::Image class and into the Scene serialization mechanism?

cdcseacave commented 5 years ago

Yes, pls add it. Starting with today I'll have some time look into the code you requested the other day. Is it still needed or better after you include domset? By the way, pls note that the original domset lib has some bugs, I remember I created some PR but the author didn't reply to this day, so I didn't report the rest of bugs (now I do not remember them).

rjanvier commented 5 years ago

Thanks. For the code review, there is no hurry. Until now it was just some tests on the SVM classification, so maybe you can have a look if you want to check if I have not made big mistakes but it's far for being crucial for now. I will begin by a PR on the "ID thing" and then the Domeset integration and so on... Regarding the Domset lib bugs, the version integrated in OpenMVG is already exempt of one major bug. At first glance it seems ok just with this fix but I will try to integrate your fix as well.

cdcseacave commented 3 years ago

The latest develop has now all the functionality needed to cluster the scene and ensure limited memory usage. Starting from a SfM solution stored into a MVS project accompanied by the undistorted images, the fist step is to compute all depth maps without fusion (the example included here uses sheffield_cross dataset):

    DensifyPointCloud scene.mvs --max-resolution 800 --fusion-mode 1 --max-threads 4 -w "sheffield_cross/opensfm/undistorted/openmvs/depthmaps" -v 0

Next split the scene in sub-scenes using the area parameter, which is something related to the inverse of GSD; it is a bit non intuitive, but normally it should remain constant for a desired memory limit; for example you can use the bellow value to limit memory usage to ~16GB:

    DensifyPointCloud scene.mvs --sub-scene-area 660000 --max-threads 4 -w "sheffield_cross/opensfm/undistorted/openmvs/depthmaps" -v 0

Disable depth-maps re-filtering by creating a file Densify.ini with just these lines:

Optimize = 0
Estimation Geometric Iters = 0

and call fusion on each of the sub-scenes:

    DensifyPointCloud scene_0000.mvs --dense-config-file Densify.ini --max-resolution 800 --number-views-fuse 2 --max-threads 4 -w "sheffield_cross/opensfm/undistorted/openmvs/depthmaps" -v 0
    ............
    DensifyPointCloud scene_000n.mvs --dense-config-file Densify.ini --max-resolution 800 --number-views-fuse 2 --max-threads 4 -w "sheffield_cross/opensfm/undistorted/openmvs/depthmaps" -v 0

This will generate for the example dateset 3 scenes: split

Any suggestions and bug-fixes are welcome, as always!

eriksobolt commented 3 years ago

I was trying this out on a different dataset. I am trying to reconstruct a wall instead of a drone photo.

So this was taken from the front instead of above. I noticed the split is in the wrong plane (see images). Is there a setting to fix this?

1 split mesh topview_1

2 split meshes topview_2

front view, I would like to split in this plane frontview

cdcseacave commented 3 years ago

it should work for any type of scene did you complete the dense reconstruction and the model looks bad? I am not sure I understand where is the last image you show in the first two; if u get bad reconstruction I need more data, like the dataset and images

eriksobolt commented 3 years ago

I think I asked my question wrong.

It is not that my result looks bad (this is a partial result). The ""--sub-scene-area 660000"" splits the scene as if you look from above (along the Z axis). This is what I am trying to show in the first two pictures. Picture one is scene_0000.ply, picture two is scene_0000.ply + scene_0001.ply. Both from a birds eye view.

Since I have want to reconstruct a wall I want the scene to be split from the front as in the third picture, so basically split that point cloud in halve

cdcseacave commented 3 years ago

the scene split is completely automatic and the user should not care how it is split; the split algorithm does not care about orientation of the scene, it does the split such that certain requirements are fulfilled, like maximum number of views in each split, etc

why do you care how the split is done?

eriksobolt commented 3 years ago

ok, maybe I will try and see what will be my result in the end.

But by the looks of it now I will not get 3 scenes like you have in the example above. But I just get a full scene that is less dense

eriksobolt commented 3 years ago

I think my scene is too small, tested it on 10 images. I let you know if it works on a bigger set

PaulLiuYZ commented 2 years ago

how can i join the different parts together?

yxjchly commented 1 year ago

How to calculate the parameter of "--sub-scene-area" in the case of different memory? eg: limit memory usage to 64GB or 128GB.

cdcseacave commented 1 year ago

I do not have a good heuristic yet, pls feel free to contribute but I'd try to double it, etc, linearly with the increase of memory

yxjchly commented 1 year ago

Tks. I have tested the value of "--sub-scene-area" in the case of memory is 128GB and "--max-resolution" is set to 3648, The maximum value of "--sub-scene-area" is approximately 549120000, in such conditions, approximately 76GB memory is used. If the value of "--sub-scene-area" is set to 591360000, it's not enough memory to fuse depth-map.

yxjchly commented 1 year ago

It doesn't seem to be a linear relationship between the value of "--sub-scene-area" and the memory usage. In the case of memory is 128GB and "--max-resolution" is set to 3648, "--sub-scene-area" is set to 549120000, approximately 76GB memory is used, it's the same with memory requirements for "--sub-scene-area" is set to 506880000. Similarly, it's the same with memory requirements for "--sub-scene-area" is set to 591360000 or 675840000.

cdcseacave commented 1 year ago

cool experiment and animation however this is similar to the normal fusion, right?

cdcseacave / openMVS

Scalable pipeline #438

sparse cloud

mesh (from from resolution-level:3)