Colmap dataset missing `get_valid_frame_id`

[pablovela5620]

Looks like the COLMAP dataset is missing the get_valid_frame_path function, I've managed to get things working on the 7scene dataset. I noticed it implemented it here https://github.com/nianticlabs/simplerecon/blob/86b647ac7e6b3a27d935d905361f16c3b0031455/datasets/scannet_dataset.py#L180

This seems to be missing from the Colmap dataset. Would I be able to just copy this function over?

[pablovela5620]

Would also be helpful if similar to 7scene docstring, the layout required for a COLMAP dataset was included. https://github.com/nianticlabs/simplerecon/blob/86b647ac7e6b3a27d935d905361f16c3b0031455/datasets/seven_scenes_dataset.py#L19

[mohammed-amr]

Hello hello,

Your title says get_valid_frame_id but looks like the problem is get_valid_frame_path. Thanks for pointing that out! Fixed now.

I also fixed a quick refactoring bug with loading in valid_frames for COLMAP, and included a more detailed docstring for the class to clear up any confusion.

It's worth mentioning that the model is trained with ScanNet images, so only scans with similar FOV ranges will work with these weights.

[pablovela5620]

Thank you for the quick fix, I'll give it a test. I did want to ask if dense reconstruction is required. I noticed that you require the undistorted images (I haven't used colmap too much but I believe this is only from the dense reconstruction)? Or is it possible to just use the sparse reconstruction

[pablovela5620]

I've also noticed that the tuple generation seemed fine, but somewhere when running the test script the scan folder is being missed. So I'm getting the following error FileNotFoundError: [Errno 2] No such file or directory: '/data/colmap_guitar/scan_1/undistorted/images/0100.JPG' from the following command

CUDA_VISIBLE_DEVICES=0 python test.py
--name HERO_MODEL
--output_base_path OUTPUT_PATH
 --config_file configs/models/hero_model.yaml
--load_weights_from_checkpoint weights/hero_model.ckpt
--data_config configs/data/colmap_default.yaml
--num_workers 8 
--batch_size 1
--run_fusion
--fuse_color
--depth_fuser open3d;

this is what my colmap_default.yaml file looks like

!!python/object:options.Options
dataset_path: /data/colmap_guitar
tuple_info_file_location: data_splits/colmap/dvmvs_split/
dataset_scan_split_file: data_splits/colmap/dvmvs_split/colmap_test.txt
dataset: colmap
mv_tuple_file_suffix: _eight_view_deepvmvs.txt
num_images_in_tuple: 8
frame_tuple_type: default
split: test

it also seems to assume a .jpg file extension (which is fine, I can convert over to that. the original files where pngs)

[mohammed-amr]

Hello! Sorry for the period of no response. I hope the problem is resolved. My last push should have fixed a few issues with the COLMAP dataset.

On my next pass I'll make it so that the extension is either read from the images file or just globs image formats from the diretory.

Let me know if there's anything that isn't working.

[pablovela5620]

Appreciate it, haven't yet managed to get COLMAP dataset to work, but I think that may have to do with the image sequence I used. Do you happen to have an example dataset similar to what was uploaded for VDR? That may help me figure out what I'm doing wrong

[pablovela5620]

I also wanted to add that the https://github.com/nianticlabs/simplerecon/blob/2ed7ec530340679d37681f9a7f8c8a1ea1d27292/datasets/colmap_dataset.py#L28 file path says to have scans as the top level for the file path, but https://github.com/nianticlabs/simplerecon/blob/2ed7ec530340679d37681f9a7f8c8a1ea1d27292/datasets/colmap_dataset.py#L505 does not include it. Same when loading the intrinsic https://github.com/nianticlabs/simplerecon/blob/2ed7ec530340679d37681f9a7f8c8a1ea1d27292/datasets/colmap_dataset.py#L291

[pablovela5620]

I managed to get things working, but I also noticed that the reconstruction visualization is upside down. The depth/normal visualization seem to be fine

https://user-images.githubusercontent.com/25287427/191612753-711651a9-56f9-44c8-bb59-7e5cc84f3137.mp4

https://user-images.githubusercontent.com/25287427/191612790-597e3c7e-e307-4118-a6d5-d3b863f35012.mp4

Lastly it seems like COLMAP generated multiple camera/images folders rather than a single one and I'm not fully sure how to connect them up

[mohammed-amr]

Thanks for trailblazing through with this! I've fixed the change to the dataset directory in all code paths that point to the scene's data in the colmap dataloader.

Great! Glad you got it working. I would guess that you have multiple disjoint COLMAP reconstructions where it couldn't link between different parts of the sequence when estimating poses.

A few quick points:

• You should probably try running with more frames in the sequence (up the frame rate).
• Looks like there isn't a good scale estimate from COLMAP for intrinsics.
• The flip is because the repo is transforming your coordinate system to ScanNet's (what the model was trained with).

Thanks!

[pablovela5620]

Appreciate the tips, looks like the latest updated didn't quite fix, I had to remove sparse here https://github.com/nianticlabs/simplerecon/blob/55ee66d40e83cb76a7bf3c5a6e9517400168b572/datasets/colmap_dataset.py#L508

as it was assuming that the images where under the sparse folder rather than the scan folder. Otherwise things seem to work okay.

The biggest problem I'm seeing (when looking at using random internet videos to generate depthmaps) is that COLMAP is exceptionally slow. For a 30 second clip at frame rate of 24FPS, it takes my machine around 30 minutes to generate the intrinsics and camera poses (this is running only the sparse reconstruction). If I'm trying to do something like a 10minute video this seems unscalable.

I was thinking that a potential solution could be to estimate camera intrinsics with COLMAP for a 30 second clip at around 5FPS and then use visual slam/odemetry software like DPVO or DroidSlam.

I'll see if that could be a viable solution

[mohammed-amr]

Make sure you're using undistorted images from colmap. That's important.

Indeed, what you suggested could work. I think you can register all the images in between 5fps with colmap. Might be faster? I haven't tried that avenue myself but have read others that have had success.