irolaina
commented
6 years ago Hi, thanks for pointing this one out!
We did indeed get the errors that we have reported in the paper. This is what we see in our machines, using Matlab version 2017a or older:
However, we looked more into this matter and came across the following problems:
-
All operations were performed in single precision (as both the network output and the depth ground truth came in single precision). However, as we sum and divide over all pixels, it turns out that their number is way too large to be handled by single precision and this leads to precision errors. We would like to report that correct evaluation requires double precision and thus we will update all reported errors (rel: 0.140, rmse: 0.597). This does not affect the delta measures.
-
This understanding came when we tested the predictions with Matlab 2017b. With this version, it looks like all operations are internally performed with double precision, even when the matrices are provided as single. Therefore, using our code in 2017b one can get accurate results.
-
In general, we observed that different versions of Matlab might cause different results, especially on different machines. It depends on different factors, such as 32-bit/64-bit, the C compiler used, etc. Similar issues have been reported: e.g. https://uk.mathworks.com/matlabcentral/answers/438-why-are-computational-results-in-matlab-sometimes-different-on-different-machines-and-how-can-i-prev
-
There might also be small differences in the output between python/matlab scripts, because of the above.
-
While investigating this, we also realized that there are perhaps small differences in how different methods evaluate their predicted depth maps. We believe that most methods use a certain center crop (as suggested by the NYU Depth dataset) in which the depth values are valid. This generally reduces errors (rel: 0.136, rmse: 0.584 in double precision). For completeness and fairness of comparisons, we will now report errors using this evaluation protocol as well.
-
We will of course update accordingly the error metrics calculation script as soon as possible.
waleedka
Hello, it is a great work. But I just test your predicted map provided by yourself and cannot get same error like your paper, I test it with matlab. your paper error: rmse 0.573 rel 0.127 your predicted map error: rmse 0.58912 rel 0.136271 Then, I get the predicted map with your pretrained model provided in this github, and write a python script to test error, I also cannot reach the results like your paper. predicted map error with your ckpt model: rmse 0.587333 rel 0.141360 I am very confused with it, could you explain it, thanks!