search

uber-research / DeepPruner

DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (ICCV 2019)
Other
354 stars 41 forks source link

Hello@uber-research, author，I would like to ask you about Kitti's evaluation~~~ #9

Closed tjutzdd closed 4 years ago

tjutzdd commented 4 years ago

How do you do? Excuse me for using poor English to consult you.

  1. For the evaluation of scene flow data set, EPE (also known as MAE) is generally used as the evaluation standard, and the evaluation function can also be implemented in the code for evaluation.

  2. For the kitti2012 data set, the evaluation criteria are > 2px, > 3px, > 4PX, > 5px of Noc and Occ (all), and the evaluation of error rate and error pixel number of mean error. Do these evaluations need to implement their functions in their own code? Or do you need to submit it to the official website of Kitti to generate the evaluation results?

  3. For the Kitti 2015 data set, the evaluation criteria include the error rate evaluation of all (Occ) and Noc's D1-bg, D1-fg and D1-all. Do you need to implement the evaluation function in the code, or must you submit the evaluation results to the Kitti official website?

  4. For kitti12, the evaluation standard can be implemented by its own code; but for kitti2015, it can't be implemented by itself. How to implement D1-bg, D1-fg and D1-all?

  5. Besides, do the experimental data of Kitti 12 and 15 have to come from Kitti's official website?

For the above problems, it's still confusing at present. Kitti website seems to say that it can't be used for debugging programs. Everyone can only submit it once within the specified time and can't apply for multiple accounts.

So for these evaluation criteria, I hope you can spare time in your busy schedule, thank you very much!!! ~~~~~~~~~~Xiaobai thanks first!!!!!!!

youmi-zym commented 4 years ago

I think the metrics including EPE, > 2px, > 3px, > 4px, > 5px can be implemented by yourself, and many released code has implemented. As for D1-bg, D1-fg and D1-all, you can also implement by yourself, as the released KITTI 2015 dataset includes a directory named 'obj_map', which is exactly the foreground, I guess.

The experimental data of Kitti 12 and 15 have to come from Kitti's official website. Yes!

KITTI is too small for the deep model's finetuning. The result of the training dataset is often unreliable and performs quite differently on the testing dataset because of overfitting. Therefore, submitting to the website is perhaps the only way to evaluate your model. Although it's kind of cruel.

ShivamDuggal4 commented 4 years ago

Thanks a lot @youmi-zym for answering the doubts of @tjutzdd

Closing this issue now. @tjutzdd Please feel free to reopen the issue in case you have any other similar doubts. I would try to answer those asap.