search

HumamAlwassel / DETAD

Diagnosing Error in Temporal Action Detectors (ECCV 2018)
http://humamalwassel.com/publication/detad/
MIT License
69 stars 21 forks source link

Fix mismatch between colors and error types #8

Closed sunniesuhyoung closed 3 years ago

sunniesuhyoung commented 3 years ago

Currently there's a mismatch between colors and error types in the right "Removing Error Impact" plot. This fix correctly matches: light blue (background), light green (double detection), blue (wrong label), red (confusion) and pink (localization).

HumamAlwassel commented 3 years ago

Hello Sunnie,

Thanks for your interest in DETAD.

I reviewed your pull request and I verified that there's no mismatch between the colors and the error types in our code. To see that, run the validation example as detailed in the README.md. With your code changes, you will see that the largest bar (middle bar) on the right changes to dark blue (i.e. Wrong Label Error). However, there are no wrong label errors in the prediction file since there are no dark blue areas in the false-positive profile on the left. Thus, we expect no impact from removing the wrong label errors (which do not exists), and the dark blue bar should be 0.0 on the right plot.

Hope this helps clear up any confusion.

Cheers!

sunniesuhyoung commented 3 years ago

Hi Humam,

I'm getting different results which I suspect to be due to some differences between Python 2 and Python 3. FYI I'm running the code in Python 3 with minor changes like replacing print '...' with print('...') and iteritems() with items().

Suppose I have:

I get the following when I run the code:

np.array(list(fp_error_analysis.average_mAP_gain.keys()))[order] [Background Err, Double Detection Err, Wrong Label Err, Confusion Err, Localization Err]

np.array(colors[1:])[order] ['#a6cee3', '#b2df8a', '#1f78b4', '#e31a1c', '#fb9a99'] = [light blue, light green, blue, red, pink]

colors[::-1] ['#a6cee3', '#e31a1c', '#fb9a99', '#1f78b4', '#b2df8a', '#33a02c'] = [light blue, red, pink, blue, light green, green]

Which is why I proposed changing colors[::-1] to np.array(colors[1:])[order], but it seems from your reply this change is inconsistent with the error type ordering in Python 2. Thanks for looking into it!

HumamAlwassel commented 3 years ago

I see! I was testing the code environment we provide in the repo (which is in Python 2). Can you please test the code on your side using the conda environment?

I also opened an issue for updating the code to Python 3. I will work on it soon. Thanks for being this to our attention.

Cheers

HumamAlwassel commented 2 years ago

Hi @sunniesuhyoung,

I have upgraded the DETAD code to Python 3. There were more issues (other than the one you pointed out) that needed fixing when switching from Python 2 to 3. So please pull the latest version.

Thanks again for bringing this to our attention :)

Cheers!