Question regarding overfit

[inkyusa]

Thank you for the interesting work. I am just wondering if you experienced any overfit while training your diffusion depth?

I tested your KITTI pipeline as described in the readme and observed that RMSE train: 0.8404 val: 6.074 test: 5.693 The command lines and hyperparams remain the same as stated in this repo.

I may make a mistake while generating dataset that is one of the differences together with different GPU devices but I think Table 2. (Quantitative depth comparison on online-KITTI dataset results) also points to the same direction as what I experienced. (summary: DiffusionDepth is 11.51 and VA-Depth 9.84 of online KITTI evaluation for SILog metric which is the lower is better while DiffusionDepth outperforms VA-Ddepth for offline as shown in Table 1.).

If you have any ideas or suggestions, it would be useful. Cheers,

[duanyiqun]

Hi, thank you very much for the interests.

Using generative model did involves more overfitting, we found it as well, but it is not as severe as you observed. I do couple of reproduce experiments have not encountered this issue. Normally for RMSE train: 0.8404 you will got val: 1.1xx test: 1.3xx. The validation curve should be like this validation curves green should be swin train from scratch, red should be swin continuous fine-tuning, green should be ResNet based.

Maybe we can check two probabilities here:

1. Have you tried the checkpoint swin to see the results? If results is ok, I guess it should be patch augmentation problem.

2. Have you checked the patching method? Please always have --patch_height 352 which means no augmentation in vertical. May you change the --patch_weight to 906 and see the result?

For the online benchmark, we boost the performance to 9.85 SILog recently. The overfitting problem always exists if we want to introduce generative methods, may need take more care of training process. I will try to increase the training stability recently.

Cheers