Problem with scalable pipeline

[lbrianza]

Hello,

I am trying to use the scalable pipeline to reconstruct a large scene, using the instructions here (which are the same as in the MvsScalablePipeline.py script). However, I must be doing something wrong, because after splitting the scene (into 2 chunks in this specific case), the densification step only produces a sparse point cloud on each of the 2 sub-scenes, and so I cannot proceed further with the mesh reconstruction.

I start from a mvs/ folder containing the undistorted images and the scene.mvs file. This one is fine, because it is produced using the main pipeline, MvgMvsPipeline.py, which is able to reconstruct the final mesh as expected, if processing all images together without splitting the scene.

However, when I split the scene, this is what happens:

1) Generating depth maps

/home/luca/openMVS_build/bin/DensifyPointCloud scene.mvs -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/ --fusion-mode 1
09:26:06 [App     ] Build date: Feb 13 2023, 14:25:26
09:26:06 [App     ] CPU: Intel(R) Xeon(R) CPU @ 2.30GHz (8 cores)
09:26:06 [App     ] RAM: 51.10GB Physical Memory 0B Virtual Memory
09:26:06 [App     ] OS: Linux 4.19.0-24-cloud-amd64 (x86_64)
09:26:06 [App     ] Disk: 134.50GB (195.80GB) space
09:26:06 [App     ] SSE & AVX compatible CPU & OS detected
09:26:06 [App     ] Command line: DensifyPointCloud scene.mvs -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/ --fusion-mode 1
09:26:07 [App     ] CUDA device 0 initialized: Tesla T4 (compute capability 7.5; memory 14.76GB)
09:26:07 [App     ] Camera model loaded: platform 0; camera  0; f 0.844x0.844; poses 328
09:26:07 [App     ] Scene loaded from interface format (294ms):
        328 images (328 calibrated) with a total of 648.63 MPixels (1.98 MPixels/image)
        266877 points, 0 vertices, 0 faces
09:26:07 [App     ] Found a camera not pointing towards the scene center; the scene will be considered unbounded (no ROI)
09:26:07 [App     ] Point-cloud composed of 266877 points with:
 - visibility info (803353 views - 3.01 views/point):
                0 points with 1- views (0.00%)
           147932 points with 2  views (55.43%)
            60750 points with 3  views (22.76%)
            58195 points with 4+ views (21.81%)
        2 min / 3.0102 mean (1.94364 std) / 58 max
09:26:09 [App     ] Preparing images for dense reconstruction completed: 328 images (1s683ms)
09:26:09 [App     ] Selecting images for dense reconstruction completed: 328 images (439ms)
Estimated depth-maps 328 (100%, 2m14s693ms)           
Geometric-consistent estimated depth-maps 328 (100%, 1m1s785ms)           
Geometric-consistent estimated depth-maps 328 (100%, 1m5s221ms)           
Filtered depth-maps 328 (100%, 58s367ms)           
09:31:31 [App     ] Densifying point-cloud completed: 266877 points (5m23s615ms)
09:31:31 [App     ] Scene saved (432ms):
        328 images (328 calibrated)
        266877 points, 0 vertices, 0 faces
09:31:31 [App     ] Point-cloud saved: 266877 points (83ms)
09:31:31 [App     ] MEMORYINFO: {
09:31:31 [App     ]     VmPeak: 14469504 kB
09:31:31 [App     ]     VmSize: 13165948 kB
09:31:31 [App     ] } ENDINFO

2) Splitting the scene

/home/luca/openMVS_build/bin/DensifyPointCloud scene.mvs -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs --sub-scene-area 10000000
09:32:50 [App     ] Build date: Feb 13 2023, 14:25:26
09:32:50 [App     ] CPU: Intel(R) Xeon(R) CPU @ 2.30GHz (8 cores)
09:32:50 [App     ] RAM: 51.10GB Physical Memory 0B Virtual Memory
09:32:50 [App     ] OS: Linux 4.19.0-24-cloud-amd64 (x86_64)
09:32:50 [App     ] Disk: 130.69GB (195.80GB) space
09:32:50 [App     ] SSE & AVX compatible CPU & OS detected
09:32:50 [App     ] Command line: DensifyPointCloud scene.mvs -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs --sub-scene-area 10000000
09:32:51 [App     ] CUDA device 0 initialized: Tesla T4 (compute capability 7.5; memory 14.76GB)
09:32:51 [App     ] Camera model loaded: platform 0; camera  0; f 0.844x0.844; poses 328
09:32:51 [App     ] Scene loaded from interface format (326ms):
        328 images (328 calibrated) with a total of 648.63 MPixels (1.98 MPixels/image)
        266877 points, 0 vertices, 0 faces
09:32:51 [App     ] Found a camera not pointing towards the scene center; the scene will be considered unbounded (no ROI)
09:32:51 [App     ] Point-cloud saved: 8 points (0ms)
09:32:52 [App     ] Scene split (1e+07 max-area): 2 chunks (610ms)
09:32:52 [App     ] Scene saved (197ms):
        147 images (147 calibrated)
        126270 points, 0 vertices, 0 faces
09:32:52 [App     ] Scene saved (239ms):
        199 images (199 calibrated)
        158019 points, 0 vertices, 0 faces
09:32:52 [App     ] MEMORYINFO: {
09:32:52 [App     ]     VmPeak: 10613548 kB
09:32:52 [App     ]     VmSize: 10580328 kB
09:32:52 [App     ] } ENDINFO

The scene seems to be split correctly into 2 sub-scenes, 1st one with 126k points, 2nd one with 158k points. All good till here.

3) Creating a Densify.ini file and adding the following 2 lines:

Optimize = 0
Estimation Geometric Iters = 0

4) Running DensifyPointCloud on both sub-scenes:

python3 /home/openMVS/MvsScalablePipeline.py DensifyPointCloud /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/ --number-views-fuse 2 --resolution-level 0 -v 2 --dense-config-file /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/Densify.ini -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/
# Module DensifyPointCloud start #
/mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/scene_[0-9][0-9][0-9][0-9].mvs
/mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/scene_0000.mvs
# Process: scene_0000.mvs
Cmd: /home/luca/openMVS_build/bin/DensifyPointCloud scene_0000.mvs --number-views-fuse 2 --resolution-level 0 -v 2 --dense-config-file /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/Densify.ini -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/
09:35:58 [App     ] Build date: Feb 13 2023, 14:25:26
09:35:58 [App     ] CPU: Intel(R) Xeon(R) CPU @ 2.30GHz (8 cores)
09:35:58 [App     ] RAM: 51.10GB Physical Memory 0B Virtual Memory
09:35:58 [App     ] OS: Linux 4.19.0-24-cloud-amd64 (x86_64)
09:35:58 [App     ] Disk: 130.69GB (195.80GB) space
09:35:58 [App     ] SSE & AVX compatible CPU & OS detected
09:35:58 [App     ] Command line: DensifyPointCloud scene_0000.mvs --number-views-fuse 2 --resolution-level 0 -v 2 --dense-config-file /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/Densify.ini -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/
09:35:59 [App     ] CUDA device 0 initialized: Tesla T4 (compute capability 7.5; memory 14.76GB)
09:36:00 [App     ] Scene loaded (245ms):
        147 images (147 calibrated) with a total of 290.70 MPixels (1.98 MPixels/image)
        126270 points, 0 vertices, 0 faces
09:36:00 [App     ] Point-cloud composed of 126270 points with:
 - points info:
        118860 points inside ROI (94.13%)
         inside ROI track length: 2 min / 2.93308 mean (1.71856 std) / 41 max
        outside ROI track length: 2 min / 3.69325 mean (2.42523 std) / 23 max
 - visibility info (375993 views - 2.98 views/point):
                0 points with 1- views (0.00%)
            69526 points with 2  views (55.06%)
            29501 points with 3  views (23.36%)
            27243 points with 4+ views (21.58%)
        2 min / 2.97769 mean (1.77686 std) / 41 max
09:36:01 [App     ] Preparing images for dense reconstruction completed: 147 images (984ms)
09:36:01 [App     ] Selecting images for dense reconstruction completed: 147 images (182ms)
Geometric-consistent estimated depth-maps 147 (100%, 1m29s826ms)           
Geometric-consistent estimated depth-maps 147 (100%, 1m34s304ms)           
Filtered depth-maps 147 (100%, 14s630ms)           
Fused depth-maps 147 (100%, 537ms)                
09:39:23 [App     ] Depth-maps fused and filtered: 147 depth-maps, 15735 depths, 1495 points (10%%) (1s660ms)
09:39:23 [App     ] Point-cloud trimmed to ROI: 1470 points removed (0ms)
09:39:23 [App     ] Densifying point-cloud completed: 25 points (3m23s243ms)
09:39:23 [App     ] Scene saved (12ms):
        147 images (147 calibrated)
        25 points, 0 vertices, 0 faces
09:39:23 [App     ] Point-cloud saved: 25 points (0ms)
09:39:23 [App     ] MEMORYINFO: {
09:39:23 [App     ]     VmPeak: 16414676 kB
09:39:23 [App     ]     VmSize: 13651044 kB
09:39:23 [App     ] } ENDINFO
/mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/scene_0001.mvs
# Process: scene_0001.mvs
Cmd: /home/luca/openMVS_build/bin/DensifyPointCloud scene_0001.mvs --number-views-fuse 2 --resolution-level 0 -v 2 --dense-config-file /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/Densify.ini -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/
09:39:24 [App     ] Build date: Feb 13 2023, 14:25:26
09:39:24 [App     ] CPU: Intel(R) Xeon(R) CPU @ 2.30GHz (8 cores)
09:39:24 [App     ] RAM: 51.10GB Physical Memory 0B Virtual Memory
09:39:24 [App     ] OS: Linux 4.19.0-24-cloud-amd64 (x86_64)
09:39:24 [App     ] Disk: 130.69GB (195.80GB) space
09:39:24 [App     ] SSE & AVX compatible CPU & OS detected
09:39:24 [App     ] Command line: DensifyPointCloud scene_0001.mvs --number-views-fuse 2 --resolution-level 0 -v 2 --dense-config-file /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/Densify.ini -w /mnt/media/tmp_reco/luca/PARKING_SCALABLE/mvs/
09:39:25 [App     ] CUDA device 0 initialized: Tesla T4 (compute capability 7.5; memory 14.76GB)
09:39:25 [App     ] Scene loaded (286ms):
        199 images (199 calibrated) with a total of 393.53 MPixels (1.98 MPixels/image)
        158019 points, 0 vertices, 0 faces
09:39:25 [App     ] Point-cloud composed of 158019 points with:
 - points info:
        145185 points inside ROI (91.88%)
         inside ROI track length: 2 min / 3.01051 mean (1.92898 std) / 44 max
        outside ROI track length: 2 min / 3.02587 mean (2.13352 std) / 28 max
 - visibility info (475915 views - 3.01 views/point):
                0 points with 1- views (0.00%)
            87491 points with 2  views (55.37%)
            36097 points with 3  views (22.84%)
            34431 points with 4+ views (21.79%)
        2 min / 3.01176 mean (1.9464 std) / 44 max
09:39:27 [App     ] Preparing images for dense reconstruction completed: 199 images (1s663ms)
09:39:27 [App     ] Selecting images for dense reconstruction completed: 199 images (187ms)
Geometric-consistent estimated depth-maps 199 (100%, 2m3s664ms)           
Geometric-consistent estimated depth-maps 199 (100%, 2m7s933ms)           
Filtered depth-maps 199 (100%, 21s124ms)           
Fused depth-maps 199 (100%, 790ms)                
09:44:04 [App     ] Depth-maps fused and filtered: 199 depth-maps, 10580 depths, 230 points (2%%) (2s357ms)
09:44:04 [App     ] Point-cloud trimmed to ROI: 0 points removed (0ms)
09:44:04 [App     ] Densifying point-cloud completed: 230 points (4m39s154ms)
09:44:04 [App     ] Scene saved (12ms):
        199 images (199 calibrated)
        230 points, 0 vertices, 0 faces
09:44:04 [App     ] Point-cloud saved: 230 points (0ms)
09:44:04 [App     ] MEMORYINFO: {
09:44:04 [App     ]     VmPeak: 17840784 kB
09:44:04 [App     ]     VmSize: 14348840 kB
09:44:04 [App     ] } ENDINFO
# Module DensifyPointCloud end #

This is where the issues occur. The two scene_dense_XXXX produced from scene_XXXX make no sense - they only contain 25 and 230 points, respectively, out of the initial 200k+ of the beginning.

If I open scene_dense.ply (produced via the "normal" pipeline, without splitting the scene) with meshlab, I see the correct pointcloud with 200k+ points. However if I open the two scene_dense_XXXX pointclouds produced with the scalable pipeline, they only contain a few sparse points.

What is possibly going on? I get the same results if I run the scalable pipeline on other scenes. I'm sure I'm doing something wrong - but where? Some wrong options in the commands?

Thanks in advance for the help!

[ghost]

The resolution is mismatched, the depth maps were created with resolution-level 1 by default then fused with resolution-level 0. Don't set sub-scene-area too high or too low. Optimize and estimation geometric iters can now be set using the command line options geometric-iters 0 and postprocess-dmaps 0.

[lbrianza]

The resolution is mismatched, the depth maps were created with resolution-level 1 by default then fused with resolution-level 0. Don't set sub-scene-area too high or too low. Optimize and estimation geometric iters can now be set using the command line options geometric-iters 0 and postprocess-dmaps 0.

Thanks for spotting the issue! I was getting crazy in trying to understand where I was doing wrong :)

[lbrianza]

I can close since the issue was solved. Thanks