Can we skip pre process in colmap if camera parameters are provided?

[henrypearce4D]

Can we skip the pre process in colmap step if the camera parameters are provided as camera_parameters.txt or even as colmap model?

I have a colmap model for my own data set, can I skip this pre process step? can the example folder structure for training be provided please?

The data structure example lists; Fabien_undist_<00257>_<08>.png This should not be ran on the undistorted but the original image - correct?

[lizhan17]

No, we cannot skip colmap step. Currently, we need colmap to generate initlized points similar to 3DGS.

Our loaders are written for the specific dataset with some hard coded name patterns. We have plan to provide custom loader.

For the technicolor, we follow previous methods that run on the undistorted images.

The format of technicolor dataset after preprocessing shoud be

<location>
|---Fabien
|   |---colmap_0
|   |    |---images
|   |    |    |---cam00.png
|   |    |    |---cam01.png
|   |    |    |---cam<>.png
|   |    |---sparse
|   |---colmap_1

the sparse point model (points3D.bin) under sparse/0 folder is loaded as the initalized points. colmap_T denotes the colmap model at timestamp T cam<M>.png means the Mth cameras we have.

Note that the png name(cam<M>.png) should be same as the extr.name in the colmap sparse model. So you need to rename your images before running colmap or you can modify the loader to make them match. image_path = os.path.join(images_folder, os.path.basename(extr.name))

[henrypearce4D]

Thanks for the information. Sorry I should have made myself clearer - I already have my own dataset with colmap initlized points similar for 3DGS for our sequences, so that's why I wanted to know if it can be skipped.

Do you have an ETA for the for the custom loader? In the meantime, if I match that structure and create a config .json I should be able to test my own dataset?

Can you confirm how the framerate is defined for the sequences? is that hardcoded as part of the data loader?

[lizhan17]

1. For a custom loader, we don't have a an ETA. But I am keeping working on this project. I think I can release it in the next several months.

2. I suggest that you can wait for our custom loader and our script for processing the custom videos. The reason is that the "scale" of colmap at different timesstamps/runs is not same if we run colmap without fixing/providing same extr and intr across time. I am not sure if your dataset's colmap sequences contain the same "scales" across differenet times. The results will be bad, if the internal "scale" of point cloud at timestamp is different.

3. Framerate is the default frame rate by the used videos. The frame rate is not hardcoded. just the pattern of filename is hardcoded.

[henrypearce4D]

By "scale" do you mean scene scale size, rotation and position? Our colmap model scales and positions are 100% consistent across time, the cameras alignment is re-used across all frames so is locked.

Here is an example of a "new splat per frame" performance, generated from the original INRIA code. https://www.youtube.com/embed/8rRp5MKTSgA?si=pTEnAgTnIpVSxrE9

Is there reference to the framerate in the colmap model files? or do the .mp4 videos need to be placed next to the colmap files to determine the framerate?

The technicolor dataset is a sequence of images not video, so how is the framerate determined?

thanks for answering these questions!

[lizhan17]

"scale" means that same point in real world should have same coordinates in world and camera across time.

We don't set the frame rate in the repo or colmap config. The framerate of all the three dataset we use is decided by their authors.

In theory, a higher frame rate will make the motion small between two consecutive frames and reduce the difficulty of modeling motions, which means better results.

[henrypearce4D]

Ok so our data should work well then as each file is trained this way, I should create a config to test training. We also can shoot at 60fps which should produce better results.

Would you be interested in receiving some data from us to test your work on the custom dataset loader? Do you have an email I can send a google drive link to?

Looking at the pretrained data in the viewer you provided, there is no defined frame rate playback speed option, so frames playback as fast as possible. Testing the n3d_sear_steak_lite_allcam data that was shot at 30fps and in the viewer VSYNC set on at 60hz, I can set the Playback Speed Modifier to 2 and get actual playback speed.

[lizhan17]

You can send me the link to my email account on the paper. I can't guarantee quick feedback on the custom loader. But I can have a quick look at it to see if a minor change of the existing loader can work with your data.

You can adjust the playback speed modifier to align the frame rate with the original video. The default modifier (1) does not strictly mean the same as the original video frame rate, as we did not set up the actual video frame rate in the repo.