Reproduce the result

[myutility]

          Hi,

you can either train the model yourself (option 1) or use one of the provided checkpoints directly for evaluation, skipping the training (option 2). Note that after training your model, you still need to run evaluation with your obtained checkpoint! For option 1, training is explained here. For option 2, evaluation with a given checkpoint path is explained here.

What you will get in the end is a directory with reconstructed views of test scenes. For computing the statistics, I used the src/scripts/compute_metrics.py script in combination with the evaluation config files in config/evaluation. Not that in the evaluation config file, e.g. config/evaluation/co3d_hydrant_extra.yaml, you need to specify the paths to the generated image directories from evaluation (you can comment out all other baseline methods). The example for extrapolation on CO3D hydrants would look like this: python -m src.scripts.compute_metrics evaluation=co3d_hydrant_extra

Feel free to reopen the issue in case of any further questions :)

Best Chris

Originally posted by @Chrixtar in https://github.com/Chrixtar/latentsplat/issues/7#issuecomment-2109979521 hi Chris, Thanks for the detailed reply. With your provided ckpt, I am able to reproduce the statistics of paper. However, I also tried to train the model by myself. Here is what I have done: I ran on python3 -m src.main +experiment=co3d_hydrant on 2 gpus with config files unchanged. After training is done, I took last.ckpt and ran eval for hydrant extra: And got: dist 0.266, lpips: 0.529, psnr: 13.855, ssim: 0.192. which is apparently not correct. Is there anything wrong with my settings? Thanks!

[myutility]

Sorry that I didn't find the reopen button so I started a new issue. I used two GPU to train because my GPU is relatively old and slow.

[myutility]

Hi,

I trained another two models using 2 GPU BS 1 setting and 1 GPU BS 2 setting. However the reproduced result is still much lower than the ckpt you provided.

Could you provide the config file (hyperparams) that you use to train so that I can reproduce it? Thanks

[Chrixtar]

Hi,

sorry for the late response. The config files are exactly the ones given in the repository and command python3 -m src.main +experiment=co3d_hydrant you mentioned is correct.

Have you trained for the full number of iterations (200k)?

For our experiments, we trained on 2 GPUs with batch size 2 each, so batch size 4 in total. Reducing the batch size can affect the performance, as 1) the learning rate might be suboptimal, as gradients become more noisy for smaller batch sizes (note that the learning rate for training the decoder is scaled by the batch size to account for that), 2) you need more iterations with a smaller batch size to effectively train for the same number of epochs (i.e. see the same amount of data (with repetitions)).

I will train it myself again as well to see the effect of different batch sizes and hopefully find out the reason why you cannot reproduce the results.