LingzheZhao
commented
3 months ago Sorry for the rambling. TL;DR: In the context of pose-optimizing NeRF/3D-GS, the novel-view evaluation requires an extra step after training, and I will release that script in a week (maybe). However, there are also some workarounds if you just want to see some deblurred images on the wandb.
Hi, it seems that you are running BAD-Gaussians on your custom dataset with the ImageRestorationDataparser.
FYI, we focus on two types of missions here, one is deblurring and the other is novel-view synthesis. For deblurring, ground-truth sharp images are required in the images_test folder for evaluation (written here), and that is only available with the synthetic datasets. While for the real-word datasets, we can only evaluate the novel-view synthesis.
The ImageRestorationDataparser works for the deblurring by default, so if images_test is not provided, there won't be any eval images, and the trainer will disable evaluation (written here). In this case, if you want to split the images into train & eval (i.e. novel-view synthesis), you can use the NerfstudioDataparser alternatively and pass these arguments after it, for example:
ns-train bad-gaussians --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
--data path/to/scene \
--pipeline.model.camera-optimizer.mode "cubic" \
--vis viewer+wandb \
nerfstudio-data --downscale_factor 1 \
--eval-mode='interval' --eval-interval=8 \Now it will use every $n^\textrm{th}$ frame for novel-view synthesis eval.
(Or you can copy the entire images folder and name the copy as images_test, and the ImageRestorationDataparser will treat them as ground-truth sharp images. Although the metrics are somehow meaningless, you can at least see the rendered sharp images pop up in the wandb.)
However, the evaluation metrics here should only be used as a reference and not as a final result - you will see the PSNR go up and then down. This is because the camera poses of the evaluation images comes from COLMAP, which are not accurate. While the camera poses of the training images are being optimized, the camera poses of the evaluation images are not.
(Also note that in our experiments, the novel-view images are sharp, unlike the blurry training images. If blurry images are configured to be eval images, it will not match our output in the evaluation (which will be deblurred sharp images, written here) and the metrics will be even less informative)
IMO, in a scene reconstruction pipeline, NeRF or 3D-GS, when you are optimizing the camera poses, which means the poses are not accurate, you can't use the poses from COLMAP to render the novel-view images - Instead, you should freeze the scene parameters and find the accurate pose for the novel-view eval image by optimizing it. At least, that's what we do in BAD-NeRF and BAD-Gaussians. Perhaps this is why many guys find that pose optimization decreases PSNR (https://github.com/nerfstudio-project/nerfstudio/issues/2863) and (https://github.com/nerfstudio-project/nerfstudio/pull/2891).
Unfortunately, I have little time to refactor the novel-view evaluation script at the moment as I'm working on some other projects. Right now it still looks like crap, maybe I'll refactor and release the script in a week.
P.S. I wrote the ImageRestorationDataparser at the beginning of the project, which doesn't seem to be useful now. Maybe I'll remove it later.
ichsan2895
ZcsrenlongZ
Eval images metrics does not show in
WanDB. It just shows GPU memory, Train loss, Train iter, and ETAInstallation:
Run command:
System information: