High EPE on the KITTI dataset

visinf / irr

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (CVPR 2019)

Apache License 2.0

194 stars 32 forks source link

High EPE on the KITTI dataset #42

Closed sbkim052 closed 3 years ago

sbkim052 commented 3 years ago

I ran "validation/pwcnet.sh" with the pre-trained model you had provided to test on the both Sintel and KITTI dataset. The result for Sintel was similar to the one you had reported (Sintel clean: 3.13, Sintel final: 4.xx). However, the EPE for KITTI12 and KITTI15 is really high (KITTI12: 23.xx, KITTI15: 17.xx). Is this the right result to get?

BTW, thanks for sharing this amazing work.

hurjunhwa commented 3 years ago

Hi, Yes, I just checked, and that's the right result. Apparently, the model trained on FlyingChairsOcc dataset doesn't generalize well on KITTI, even much worse than the reported number on the PWC-Net paper (EPE on KITTI15: 13.20).

sbkim052 commented 3 years ago

Thanks for your quick response:) I am training vanilla PWCnet with flyingchair at this moment.. But what you are saying is that there won't be any improvement on vanilla PWCnet in generalization if trained on only Flyingchairs?

hurjunhwa commented 3 years ago

I think it's a reproducibility issue of the vanilla PWC-Net.

Possibly due to:

Dataset bias: For KITTI, training on FlyingChairs may be better than training on FlyingChairsOcc.
my PyTorch version uses different augmentation parameters for training, comparing to the original Caffe implementation.

When the training on FlyingChairs is done, probably you can evaluate it on KITTI and see if it's the dataset bias problem..

sbkim052 commented 3 years ago

Thank you @hurjunhwa :)