Issue with Seam Artifacts in Point Cloud

[geofis]

Hello,

I'm encountering what I think is an issue with a point cloud generated using OpenDroneMap, which incorporates OpenMVS for point cloud densification. The problem arises in a photogrammetric flight dataset consisting of historical aerial photographs with generally low overlap (around 60%). A small strip within the dataset is captured in three frames, while the majority is only in two (see below). This leads to a noticeable elevation discrepancy at the seam between these areas.

I have generated a DSM (in addition to the point cloud) using OpenDroneMap, which I am including here for illustrative purposes:

Point Cloud, general view (the seam artifacts are encircled within the red outline rectangle):

Point Cloud highlighting the area with the artifact:

DSM Image (generated in ODM, for illustration only):

Number of Views Image (generated in ODM, for illustration only):

To ensure proper rendering in areas with only two views, I followed the recommendation provided in this issue. While the solution offered in that issue is nearly perfect, the mentioned seam remains a persistent artifact that I'm hoping to eliminate. As I understand it, this seam is created at the transition from the strip photographed three times to the rest of the areas photographed twice.

I've experimented with various adjustments to mitigate this artifact, such as tweaking brightness, applying white balance, utilizing grayscale (tricking OpenMVS with three-channel files generated from a grayscale source), applying blurred brightness, modifying highlights/shadows, but none have successfully removed the seam. Additionally, I added the Max Views = 2 option in the Densify.ini file to limit densification to two views, resulting in the following OpenDroneMap configuration:

Min Views Filter = 1
Max Views = 2
Optimize = 7

(The last line is automatically added by OpenDroneMap at runtime.)

Could you provide guidance on alternative approaches or settings I could try to resolve this seam issue? Any insights or suggestions would be greatly appreciated.

Thank you for your assistance.

[cdcseacave]

Usually such problems are caused by poor pose estimation during SfM. In this case this suspicion is doubled by the low image overlap. Can you try doing SfM with a different pipeline? OpenMVG, COLMAP, Metashape, ContextCapture (for all there is an importer to MVS scene)?

[geofis]

Thank you for your response. Based on your suggestion, I will attempt to conduct the SfM process using alternative pipelines (COLMAP seems pretty fine). Once I have completed these tests, I will report back with my findings.

[geofis]

Just a brief message in case someone else encounters this same issue.

The stitching artifact was, as @cdcseacave initially suggested, due to a problem in the reconstruction process. In my case, where I was trying to process historical aerial photographs in OpenDroneMap (see associated thread here), the reconstruction failed because I was not properly informing OpenSfM about the camera calibration. To resolve this, I had to construct a geo.txt file (with coordinates of the frame center) which OpenDroneMap sends to OpenSfM to use in generating the calibration and facilitating a seamless reconstruction. Another aspect that greatly affected the reconstruction was that each image file had different dimensions, making it even more difficult for OpenSfM to process. It's important, for the reconstruction phase, to use geometrically similar files and not ones deformed by the user.

Thank you, @cdcseacave, for pointing me in the right direction.

[geofis]

Apologies for reopening this issue.

Once again, thank you @cdcseacave for this efficient software.

I am reopening this issue because, following @cdcseacave's recommendation, I managed to reconstruct with COLMAP and densify with OpenMVS a scene from an area I am working on with historical aerial photographs. The workflow I followed in COLMAP was this (inspired by this issue-tutorial):

####### STEP 1 #######
colmap feature_extractor \
  --SiftExtraction.use_gpu 0 \
  --SiftExtraction.max_image_size $MAX_IMG_SIZE_EXTRACT \
  --SiftExtraction.max_num_features $MAX_NUM_FEATURES \
  --database_path $PROJECT/database.db \
  --image_path $DATA_ROOT/$PROJECT/images | tee -a $PROJECT/salida_consola.txt

####### STEP 2 #######
colmap exhaustive_matcher \
  --SiftMatching.use_gpu 0 \
  --database_path $PROJECT/database.db | tee -a $PROJECT/salida_consola.txt

####### STEP 3 #######
colmap mapper \
  --database_path $PROJECT/database.db \
  --image_path $DATA_ROOT/$PROJECT/images \
  --Mapper.tri_ignore_two_view_tracks 0 \
  --output_path $PROJECT/sparse | tee -a $PROJECT/salida_consola.txt

####### STEP 4.1 #######
colmap image_undistorter \
  --image_path $DATA_ROOT/$PROJECT/images \
  --input_path $PROJECT/sparse/0 \
  --output_path $PROJECT/dense \
  --output_type COLMAP | tee -a $PROJECT/salida_consola.txt

####### STEP 4.2 #######
colmap model_converter \
  --input_path $PROJECT/dense/sparse \
  --output_path $PROJECT/dense/sparse  \
  --output_type TXT | tee -a $PROJECT/salida_consola.txt

Then, I processed the reconstruction obtained in OpenMVS with this configuration:

####### STEP 5 #######
InterfaceCOLMAP \
  --working-folder $(pwd)/ \
  --input-file $PROJECT/dense/ \
  --output-file $PROJECT/dense/model_colmap.mvs | tee -a $PROJECT/salida_consola.txt

####### STEP 6 #######
DensifyPointCloud \
  --working-folder $(pwd)/$PROJECT/dense \
  --input-file model_colmap.mvs \
  --resolution-level 2 \
  --dense-config-file Densify.ini \
  --number-views-fuse 2 \
  --max-resolution $MAX_DIM \
  --output-file model_dense.mvs | tee -a $PROJECT/salida_consola.txt

To clarify, Densify.ini only contains this:

Min Views Filter = 1
Optimize = 7

Although I managed to slightly mitigate the vertical shifts in the point cloud (artifacts) by reconstructing with COLMAP, I couldn't completely eliminate them, as can be seen in this sample.

This is a colorizd RGB PC of the same area, for reference:

This is the sparse PC:

Note that the central frames are somewhat different from the rest (they have a slightly lower resolution, something I cannot resolve as the frames are not mine), but this should not be an issue, as I have tested with photographs that are practically identical in resolution, and yet the vertical shifts between frames still appear.

I have tested about 100 different configurations with OpenSFM+OpenMVS (see here for a more complete discussion), so it is likely that the reconstruction bears some responsibility, because using the reconstruction from COLMAP the result improved a bit, but I want to be sure that the densification isn't introducing a bias in intersected views (perhaps in the depth map fusion?). Likewise, I have tested with multiple sets of photographs, and in all cases, to varying degrees, the vertical shifts are present in the point cloud.

The set of frames with which I created the above scene can be found here, in case anyone would like to reproduce it.

Any suggestions on where I should explore further?

[cdcseacave]

If I understand right, the issue seems to be in poor pose accuracy. Did you try using OpenMVG for SfM? It is very easy, just use MvgMvsPipeline.py as instructed here.

[geofis]

Thank you. I tried to install the suite by compiling in a clean docker environment and also inside a virtual machine, but couldn't make it work. I opted for QUICK_START.sh docker solution, but it doesn't include the OpenMVG-OpenMVS pipeline. Is there a more suitable solution?

[geofis]

However, I still understand that the reconstruction doesn't seem to be inadequate, since the camera parameters and positions look coherent, and the model appears consistent, even in the intersection between disparate photographs, or in the intersection between areas of two and three views (no sharp displacement can be seen). This is a view of the model from COLMAP:

[cdcseacave]

I run SfM in metashape and then the densification in OpenMVS looks good. So for sure is the inaccuracy in the camera poses

On Sun, Apr 7, 2024 at 18:05 José Ramón Martínez Batlle < @.***> wrote:

However, I still understand that the reconstruction doesn't seem to be inadequate, since the camera parameters and positions look coherent, and the model appears consistent, even in the intersection between disparate photographs, or in the intersection between areas of two and three views (no sharp displacement can be seen). This is a view of the model from COLMAP:

image.png (view on web) https://github.com/cdcseacave/openMVS/assets/13460617/3f30d1f4-c727-4e76-bf1a-05ca18f17f15

— Reply to this email directly, view it on GitHub https://github.com/cdcseacave/openMVS/issues/1110#issuecomment-2041499368, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVMH3WL7DHSUOHUJ6XTKBLY4FOD3AVCNFSM6AAAAABDMO4AXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBRGQ4TSMZWHA . You are receiving this because you were mentioned.Message ID: @.***>

[geofis]

Thank you for your response. I'm not familiar with integrating Metashape results into OpenMVS, so I guess I will have to do some research. Any tips would be greatly appreciated.

[cdcseacave]

Pls check the wiki,I think I explained there

On Mon, Apr 8, 2024 at 16:27 José Ramón Martínez Batlle < @.***> wrote:

Thank you for your response. I'm not familiar with integrating Metashape results into OpenMVS, so I guess I will have to do some research. Any tips would be greatly appreciated.

— Reply to this email directly, view it on GitHub https://github.com/cdcseacave/openMVS/issues/1110#issuecomment-2042759603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVMH3SXV675FRZZ6S3RSBLY4KLNTAVCNFSM6AAAAABDMO4AXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBSG42TSNRQGM . You are receiving this because you were mentioned.Message ID: @.***>

[geofis]

Hello,

I've successfully managed to input reconstructions from Metashape into OpenMVS. I created OpenMVS densifications based on Metashape models, but the vertical discontinuities were still present. Here's one of the outcomes:

Reconstruction in Metashape

Point cloud generated in OpenMVS using a reconstruction from Metashape (discontinuitites persist): I'm not sure about the settings you used in Metashape, but I tried with the "Highest" accuracy and eventually set the camera calibration, ensuring the use of proper focal length and pixel size. The reconstruction itself didn't seem erroneous.

I also attempted generating the point cloud directly in Metashape, but the outcome was unsatisfactory as well, with the discontinuities still appearing. This somewhat reassures me because it suggests that if all algorithms are producing the same result, it indicates an issue with the photographs themselves. The unfortunate part is that historical aerial photographs are such rich sources, yet they seemingly can't be directly utilized in modern photogrammetric workflows "out of the box."

In this repository...

https://github.com/tudipffmgt/HAI-SFM/

... researchers have hosted a pipeline based on DISK and SuperGlueNetwork to address the reconstruction issue. I replicated their workflow with my photos, but the result was far from satisfactory.

[geofis]

Finally, Metashape successfully created a point cloud free of vertical discontinuities, achieving a good result.

Using the same Metashape reconstruction, I attempted the densification in OpenMVS, but encountered vertical discontinuities again.

While the exact source of this issue remains unclear to me, this recent test strongly suggests that the reconstruction by Metashape is quite precise, and this accuracy extends to the dense point cloud it generates. This leads me to think that the issue likely arises beyond the reconstruction phase.

However, I will try to clearly lay out the main differences between the configuration parameters used in Metashape and in OpenMVS.

[cdcseacave]

not sure about what discontinuities are you talking; I did a densification in OpenMVS, see bellow, and the geometry is continuous: and from side:

As you can see the geometry is continuous, there is only a difference in color (to be exected as at this point there is no blending), and in density of the point, as the density is directly related to the resolution of hte images

[geofis]

Thank you for taking the time to try and reproduce this issue.

To reproduce the vertical discontinuities, please try the following:

1. Load the point cloud into a viewer that allows you to level and orient it according to the proper XYZ axes, ensuring that Z is truly the vertical axis (e.g., terrain elevation). In my case, I use CloudCompare, under the option Edit > Translate/Rotate.

2. Zoom in closely on the junction bands between frames (the central ones are the best in the example data), which are noticeable due to chromatic differences. These often coincide with the intersection lines between areas of two and three views and usually have slightly different point densities.

3. Apply a vertical exaggeration, which in the case of CloudCompare is usually achieved with Edit > Multiply/Scale. Apply a factor of 3 only to the Z-axis. The jumps should become more visible.

4. To improve the visualization of the discontinuities, it's preferable to use Z as a scalar field and apply a color palette by shrinking the visualization table (I limit the lookup table to the values where the discontinuities lies). Normally, in each test, I used to remove noise (I don't do it in the latest tests due to exhaustion), which helped not to be distracted by disparate points, but this is not essential.

I have created point clouds from frames that appear to be perfectly scanned and have uniform resolution, supplied directly with minimal manipulation on my side, yet the issue of discontinuities continues to persist in them. I have conducted many tests (about 100), and I keep the commands and resulting point clouds that I use in each case (e.g., processing workflows with ODM, COLMAP+OpenMVS, Metashape+OpenMVS), which I can share. In all the cases, the vertical discontinuities persists.

You can find more information in this post on the OpenDroneMap forum.