Reproducing RMSE result on "Original" images

DeepInsight-PCALab / ST-CGAN

Dataset and Code for our CVPR'18 paper ST-CGAN: "Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal"

71 stars 9 forks source link

Reproducing RMSE result on "Original" images #6

Closed LetiP closed 5 years ago

LetiP commented 5 years ago

Hello! I am trying hard to reproduce your baseline, so your "Original" RMSE values of 32.67 on shadow regions, 6.83 on non-shadow, 10.97 on all.

I downloaded your ISTD dataset from Google Drive, loaded the test images test_A and test_C and computed RMSE on them after converting them to LAB color space.

Why do I get values like 6.9 for "all" regions and not anything near your 10.97? Is there anything I do not understand from your paper?

Thanks in advance!

LetiP commented 5 years ago

Update: I get closer to your numbers, if I compute the RMSE in percentage on the RGB, not LAB space: So: shadow: 22.8% non-shadow: 6.2% all: 10.5%

It is still a mistery to me how to get the right numbers. Since it is only a computation on the original dataset and not a method evaluation.

csjfwang commented 5 years ago

Dear @LetiP, Sorry to reply a little late. It may be that the way we calculate the RMSE error is different, so our results are different.

The evaluation method and code we use are the same as that of GUO's CVPR11 paper. You can email me (jfwang.cs at gmail.com). And I will provide you with the original evaluation code.

Thanks! Jifeng.

LetiP commented 5 years ago

Thanks again for the code and for the quick response!

Perhaps as a clarification for others: In my opinion, the paper used not the RMSE (root mean squared error), but the MAE (mean absolute error). With the MAE, I am able to reproduce the results. :)