issue about evaluation

[Torment123]

Hi, when testing the trained models, I found that there is one example in the test set of shapenet cars - 876d92ce6a0e4bf399588eee976baae, that only contains white backgrounds, and somehow isn't filtered out in the eval.py process resulting in a pseudo high psnr result. After removing this example, the cars checkpoint provided in the repo has a test psnr of 23.93, just want to confirm this, thanks.

[szymanowiczs]

I checked that indeed this test example contains white backgrounds. This test dataset is a standard benchmark across many works, so I used it as-is, without any filtering, to be comparable with prior works. When making comparisons you probably should not exclude this example, or you should exclude it for all methods.

Since the example is backgrounds only, it's possible that for this example PSNR is higher than for dataset average, and that removing it worsens the average score. Not sure if it's 23.93 but that sounds plausible.

[Torment123]

I checked that indeed this test example contains white backgrounds. This test dataset is a standard benchmark across many works, so I used it as-is, without any filtering, to be comparable with prior works. When making comparisons you probably should not exclude this example, or you should exclude it for all methods.

Since the example is backgrounds only, it's possible that for this example PSNR is higher than for dataset average, and that removing it worsens the average score. Not sure if it's 23.93 but that sounds plausible.

Thanks for the clarification. However in line 115-116 in eval.py, there is code that excludes non-foreground images for metric computation, but it only excludes those with black background, I wonder whether it's intended like this so the white background scene can be kept, or is it a bug?

[szymanowiczs]

This is intentional - the exclusion is because for CO3D it's a custom split and I decided to exclude that example (both for my method and the baseline). Hope this clarifies things.