How do you evaluate the kitti eigen split?

[xingyi-li]

Hi, I appreciate your work a lot, thanks for releasing the code.

In your code evaluate.py, I see that you do Grag crop to depth_gt, but I also see that in your test dataloader, when you load the depth_gt, you do kitti benchmark crop, i.e., crop to (1216 x 352). Hence, you first crop the depth_gt to size of (1216 x 352), and then you crop them according to Grag crop. Did I miss something? Is that true? Can you explain the reasonability?

By the way, do you evaluate the kitti eigen split on 652 test images or on the raw 697 test images?

Thanks again!

[shariqfarooq123]

The paper uses Garg crop as defined here. All the crops are optional (technically) and you can easily enable or disable them during both training or testing by passing (or not passing) the respective arguments. See train.py and evaluate.py for the arguments explanation. Note that the Garg crop is commonly used as an evaluation mask and cuts off a significant portion (~40%, mostly in the sky-region) whereas kb_crop is more like an optional preprocessing step (e.g. Official dev-kit) for stripping off invalid regions (only about ~2-6%). You can refer to previous works such as this or this for usage.

For KITTI, official annotated depth data is used for both training and testing, found here: http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_prediction

[serizba]

Hi @shariqfarooq123, Thanks for releasing the code of your work!

I've just ended with the same doubt as @xingyi-li. You pointed to previous work BTS where they do the kb_crop also for evaluating. I've just realized that BTS performs the crop on the input image, but then, when evaluating they use the full, uncropped ground truth. What they do is to fill the cropped predicted depth to match the ground truth depth.

On the other hand, on your code, when evaluating you perform first the kb_crop and then the garg_crop, what means that you are evaluating with a considerable smaller portion of the ground truth. Evaluating on the ground truth only with the garg_crop (and inputs with both crops as in BTS) worsens the results.

I am missing something? or is there any other work that performs both crops on the GT?,

thanks a lot!

[surajiitd]

kb_crop is a crop.
while garg_crop/eigen_crop are just evaluation masks.