How to train on whole N3DV dataset?

[adkAurora]

Both the paper and the code train only 50 frames, how can the entire N3DV dataset of 300 frames be trained? It seems impossible to increase the durationparameter due to GPU memory limitations.

[lizhan17]

Hi, we evaluate on the 300 frames. You can shift the source_path from colmap_0 to colmap_50, colmap_100.

Besides, the memory is not limitation if we use lazy load image.

[adkAurora]

Thank you for your reply. I have another question: is the 'duration' parameter here the same as variable 's' in the temporal radial bias function described in the paper?

[lizhan17]

Hi, No, duration in config is only for video frame length we used. We have _trbf_scale and _trbf_center for the temporal radial basis function.

[minsu1206]

Hi there,

I have a question regarding training on the N3DV dataset. Allow me to clarify my understanding:

To train and evaluate with approximately 300 frames, follow these steps:

python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_0 --source_path <location>/cook_spinach/colmap_0 
python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_50 --source_path <location>/cook_spinach/colmap_50
python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_100 --source_path <location>/cook_spinach/colmap_100
python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_150 --source_path <location>/cook_spinach/colmap_150 
python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_200 --source_path <location>/cook_spinach/colmap_200
python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite_250 --source_path <location>/cook_spinach/colmap_250

python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_0 --source_path <location>/cook_spinach/colmap_0
python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_50 --source_path <location>/cook_spinach/colmap_50
python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_100 --source_path <location>/cook_spinach/colmap_100
python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_150 --source_path <location>/cook_spinach/colmap_150
python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_200 --source_path <location>/cook_spinach/colmap_200
python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath config/n3d_<lite|full>/cook_spinach.json --model_path <path to model>_250 --source_path <location>/cook_spinach/colmap_250

After these steps, I plan to take an average of the results obtained from cook_spinach_0 through cook_spinach_250. Could you confirm if this approach is correct?

Thank you.

[lizhan17]

yes.

[minsu1206]

Thank you !

[lizhan17]

please remove here for paper (may cause some instability result of some scenes). Difference should be small.

https://github.com/oppo-us-research/SpacetimeGaussians/blob/aa5a1618842ee0d9c605e9941aa8bd95a62c578b/helper_train.py#L186

L186 to 191 , i added for our viewer's compatibility. but it comes out the problem is not the minimax of points. We already fix the viewer's compatibility

[minsu1206]

Hi

I am trying to train whole N3DV dataset following above instructions. But I often meet error :

Traceback (most recent call last):                                                                                                                                                                                                     
  File "train.py", line 392, in <module>                                                                                                                                                                                               
    train(lp_extract, op_extract, pp_extract, args.save_iterations, args.debug_from, densify=args.densify, duration=args.duration, rgbfunction=args.rgbfunction, rdpip=args.rdpip)                                                     
  File "train.py", line 147, in train                                                                                                                                                                                                  
    depthdict[viewpoint_cam.image_name] = torch.amax(depth[slectemask]).item()                                                                                                                                                         
RuntimeError: amax(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument. 

I found that all the values of depth right before train.py line 147 are 15 and this leads to above error because depth[slectmask] is empty tensor at this case. Indeed, 15 is hand-crafted value at gaussian_rasterization_ch9/cuda_rasterizer/forward.cu line 308, so I suspect there's some problem on initial point cloud.

I tried 2 ways (1) just running again (2) preprocess again and run

These sometimes solve the problem, but still induce errors. How to solve this problem? I would appreciate your advice on this.

Thanks.

[lizhan17]

how many inital points do you have ? usually inital points for 50 frames should be 380k. or gaussians._xyz.shape[0] at the very begining.

[minsu1206]

Around 170k ~ 180k.

Then, running COLMAP with other options to get more points or hand-crafted point cloud augmentation might be helpful. How do you think about it ?

[lizhan17]

170k points for which scene and which model ?

I think colmap should produce similar number of points.

[minsu1206]

Sorry for confusing.

I use full model. To be specific regarding to scene info, here is the description

sear steak 176k ~ 179k points

cook spinach 186k ~ 189k points

flame salmon 202k ~ 203k points

cut roasted beef 164k ~ 165k points

This means COLMAP produce similar number of point per 50 frames.

[lizhan17]

I mean for n3d"s full model the initial points should be at least 380k.

you can run the pre_n3d with option --endframe 300 To extract all the points in 300 frames

[minsu1206]

Yes, I already run pre_n3d.py on whole N3DV dataset with option --endframe 300.

By the way , The numbers I described are from gaussian_model.py / def create_from_pcd / ln132 print("Number of points at initialisation : ", fused_point_cloud.shape[0]) (fetch from totalply_path)

Is what you said the same as what I wrote?

[lizhan17]

Yes it should be same as in full model we use all the points available.

[lizhan17]

I recommend that you pick a scene clean all the files, redo the whole preprocess step.

The environment should be colmapenv

[minsu1206]

Ok. I will try again. Thanks for your help.