Use MipNeRF360 data factor in evaluation.

[jefequien]

The MipNeRF360 evaluation procedure usually requires handling indoor and outdoor scenes with different data factors. Is this handled in this codebase? https://github.com/YihangChen-ee/HAC/blob/c23a4dc85edf9e5b70a7b735df284143cd477994/run_shell_mip360.py#L5

See original 3DGS evaluation. https://github.com/graphdeco-inria/gaussian-splatting/blob/8a70a8cd6f0d9c0a14f564844ead2d1147d5a7ac/full_eval.py#L41

See gsplat evaluation. https://github.com/nerfstudio-project/gsplat/blob/c14d720c561abaf6ff9ce131d284a59104dda5f5/examples/benchmarks/basic.sh#L8

[YihangChen-ee]

Hi, thanks for your question. We basically follow training scripts of Scaffold-GS to always use folder "images" to train MipNeRF360. The dataloader would automatically adjust resolutions to no larger than 1.6K. In our paper, we also use folder "images" to train 3DGS, so it is still a fair comparison with HAC.

For your request, we change image folders according to indoor and outdoor scene types, and get the following results on HAC:

PSNR | SSIM | LPIPS | SIZE(MB)

for high rate, $\lambda$=0.5e-4: 27.5938649 | 0.8086581 | 0.23386593 | 22.5002889

for low rate, $\lambda$=4e-4: 27.301666 | 0.8029547 | 0.24558464 | 14.3753333

[jefequien]

Hey! Thanks for the quick response and thank you very much for rerunning the experiments! I'll let the 3dgs compression survey people know.

[paulbzm]

Thank you @jefequien for noticing, and thanks @YihangChen-ee for providing the new numbers, they are now updated on our website! If you would like to provide the results of the remaining λ with the data factors, we could include them in our plots.

[paulbzm]

We reverted your results to the original results of your paper.

You already adhered to the testing conventions of 3DGS, as they also scale the MipNeRF360 results to the maximum width of 1600px in the evaluation.

The confusion appeared because 3DGS uses the data factors in the training, however they don't use it in the rendering of the results, as can be seen in their full_eval file. For our survey it is only necessary to follow the conventions in the evaluation.

Apologies for any inconvenience this may have caused.