search

isl-org / ZoeDepth

Metric depth estimation from a single image
MIT License
2.33k stars 214 forks source link

About evaluation on SUNRGBD dataset #19

Open whyxz opened 1 year ago

whyxz commented 1 year ago

Hello, may I ask why the evaluation metric is NaN and the error message shows "mean of empty slice" when testing on SUNRGBD dataset?

shariqfarooq123 commented 1 year ago

Hi, Thats strange, that would mean there were no valid ground truth depth values in the eigen crop for that image. Could you visualize the ground truth depth for the sample you're getting this error for?

whyxz commented 1 year ago

Thank you, I'll give it a try.

whyxz commented 1 year ago

微信图片_20230321205742 The issue seems to be here on line 96. Changing -1 to 9 will prevent NAN from occurring, but the results are not very satisfactory. I printed out the predicted results and ground truth and found a large discrepancy. 微信图片_20230321205753 微信图片_20230321205757

whyxz commented 1 year ago

The official GT provided is like this. 微信图片_20230321220219

shariqfarooq123 commented 1 year ago

Thanks for sharing the details. Can you recheck what is the min and max for this ground truth (before capping the max in line 96)?

whyxz commented 1 year ago

微信图片_20230321221542 After dividing GT by 1000, it seems that the unit is no longer meters

shariqfarooq123 commented 1 year ago

My guess is that, for some reason, your Ground Truth depths are off by a factor of 10 or so. Can you try dividing by 10000 and checking the results?

This may happen if we acquired the data from different sources or if the official dataset has changed since (format or data itself) (We have had the dataset for more than two years now).

whyxz commented 1 year ago

Thank you for reply, I'll give it a try.

whyxz commented 1 year ago

微信图片_20230322093558 Hi，I divided it by 10,000 and the result is still very strange. Could you please share your test set with me?

Twilight89 commented 1 year ago

ruth depth values in the eigen crop for that image. Could you visualize the ground truth depth for the sample you're getting

I met the same problem, maybe the author did not use the official SUNRGBD datasets from the website.