search

Cogito2012 / DEAR

[ICCV 2021 Oral] Deep Evidential Action Recognition
Apache License 2.0
122 stars 19 forks source link

About evaluation protocols. #8

Closed wjun0830 closed 1 year ago

wjun0830 commented 1 year ago

Hello.

Thanks for the interesting work.

Besides, I was curious about the design choice of the evaluation metric regarding 10 random trials of unknown class selection.

What's the difference between using a whole hmdb51/MiT-v2 dataset as open set?

Since the training is conducted with the whole UCF101 dataset, I was thinking that using hmdb51/MIT dataset as a whole would make no difference to your evaluation protocols.

Thank you for the great work.

Cogito2012 commented 1 year ago

The purpose of 10 random trials of unknown classes is to show stable/convicing evaluation result w.r.t. corresponding openness of the osr testing.

You can definitely use the whole hmdb/mit dataset as the unknown so that no random trial is needed, but note that this only reports the performance under a single openness value.

wjun0830 commented 1 year ago

I get it. Thank you.