Issue with multi-camera COLMAP dataset

[VisionaryMind]

I have successfully trained with ADOP on scenes that use a single camera. I am having difficulty, however, with a larger dataset comprised of 882 cameras, 2/3 of which are from drone footage, the other 1/3 being a hand-held DSLR.

COLMAP successfully generates the camera registrations and point cloud, and running colmap2adop correctly creates all the necessary files for training. I am noticing, however, that COLMAP resizes every single image, effectively generating 882 individual camera profiles. I thought at first this wouldn't necessarily be an issue, as adop_train can parse multiple cameras. I am running into this error, though:

Assertion 'cam.w == image_size_input(0)' failed!
  File: /home/visionarymind/Downloads/ADOP/src/lib/data/Dataset.cpp:49
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)
Aborted (core dumped)

I have looked at each camera.ini file, and the dimensions match each image. Would you have any idea why this error would be thrown and any way to work around it? Technically, there were only two cameras, but COLMAP resizes the images during registration.

[VisionaryMind]

I can confirm that ADOP does not work with multiple cameras, regardless of configuration. Over the weekend, we have tested on several large datasets, and even with only two different cameras, it fails with:

====================================
step 1
[...]
Assertion 'cam.h == image_size_input(1)' failed!
  File: /content/ADOP/src/lib/data/Dataset.cpp:50
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)

Bring the camera count down to one (i.e. exclude images with different focal lengths, even from the same camera), and the training proceeds without a problem. Please let me know where the code should be modified to allow multiple image sizes, and I will make the change accordingly and re-test. For now, this cannot be used in the majority of photogrammetric pipelines where there is more than one camera and / or images with different focal lengths / sizes. Even with a set of images all of the same size, this will be a problem because during undistortion processing, COLMAP changes image sizes.

[darglein]

• First of, you should never use the undistorted images of COLMAP (you lose sharpness if you do). Always use the original raw images if possible. ADOP directly supports the pinhole camera model with distortion.

• Can you provide me a multi camera dataset with different image sizes? I can take a look and implement an update for you :)

[VisionaryMind]

@darglein How have you been able to generate the COLMAP fused point cloud without first undistorting the images? The problem is that those undistortions and resulting image re-sizing are baked into the cameras.bin / images.bin, and if I were to simply copy over the original images alongside the COLMAP dataset, your colmap2adop routine would not be able to parse the dataset properly. I will send you a link via e-mail to the dataset (Kyrkogatan).

UPDATE: I have re-attempted running adop_train with a the COLMAP dataset, using the original undistorted images, the the same problem is present. Even with the original images, colmap2adop resizes the images and creates three cameras --- then is unable to match the resolutions.

[Gatsby23]

• First of, you should never use the undistorted images of COLMAP (you lose sharpness if you do). Always use the original raw images if possible. ADOP directly supports the pinhole camera model with distortion.
• Can you provide me a multi camera dataset with different image sizes? I can take a look and implement an update for you :)

Sorry for troubling you. I have met the same problem too. The dataset I used is https://www.cs.ubc.ca/research/kmyi_data/imw2020/TrainingData/brandenburg_gate.tar.gz. Because the dataset provide the finished spare reconstruction. So I use the undistort command and reconstruction command to reconstruct the scenes. And I use the colmap2adop to construct the scenes. However, the error shows that the image w is not as same as the camera w.

[Gatsby23]

@darglein How have you been able to generate the COLMAP fused point cloud without first undistorting the images? The problem is that those undistortions and resulting image re-sizing are baked into the cameras.bin / images.bin, and if I were to simply copy over the original images alongside the COLMAP dataset, your colmap2adop routine would not be able to parse the dataset properly. I will send you a link via e-mail to the dataset (Kyrkogatan).

UPDATE: I have re-attempted running adop_train with a the COLMAP dataset, using the original undistorted images, the the same problem is present. Even with the original images, colmap2adop resizes the images and creates three cameras --- then is unable to match the resolutions.

Have you solved the problem ? I comment the Saiga_ASSERT, but it breaks down at training.

[darglein]

Hello, the latest commit should fix this issue. I was able to train ADOP on the kyrkogatan dataset using the undistorted images.

For an even better reconstruction, I suggest using the original images (maybe downscaled by a factor of 2).

https://user-images.githubusercontent.com/16142878/178506682-4968f6b1-822b-432d-ba23-0f146243d05f.mp4

[VisionaryMind]

This is beautiful. Thank you kindly for taking the time to resolve the issue! If I understand you correctly, you are saying that reconstruction will be more accurate with 2x downsized original images? Wouldn't we be losing keypoints during matching? I have attempted a low-resolution reconstruction using other photogrammetry tools in the past, and it has always yielded less accurate results. I am curious to hear your own experience.

[Gatsby23]

This is beautiful. Thank you kindly for taking the time to resolve the issue! If I understand you correctly, you are saying that reconstruction will be more accurate with 2x downsized original images? Wouldn't we be losing keypoints during matching? I have attempted a low-resolution reconstruction using other photogrammetry tools in the past, and it has always yielded less accurate results. I am curious to hear your own experience.

I met the same problem as you. After loading the config, it stops at once. How doyou fix that ?

[VisionaryMind]

I met the same problem as you. After loading the config, it stops at once. How doyou fix that ?

In my case, I had shifted the images folder into the root ADOP dataset, thinking that it would need to look for them there. On the contrary, the images must be in the same hierarchy as they were when you run colmap2adop. Otherwise, they cannot be loaded, and it will fail.

[Gatsby23]

I met the same problem as you. After loading the config, it stops at once. How doyou fix that ?

In my case, I had shifted the images folder into the root ADOP dataset, thinking that it would need to look for them there. On the contrary, the images must be in the same hierarchy as they were when you run colmap2adop. Otherwise, they cannot be loaded, and it will fail.

You can modify the dataset.ini in the scenes folder. Thanks for your advice