search

arthurdouillard / incremental_learning.pytorch

A collection of incremental learning paper implementations including PODNet (ECCV20) and Ghost (CVPR-W21).
MIT License
383 stars 60 forks source link

Accuracy to report #51

Closed ashok-arjun closed 2 years ago

ashok-arjun commented 2 years ago

Hi @arthurdouillard ,

Thanks for the detailed display of all metrics such as a old classes, new classes - they are very helpful.

I have a question regarding the accuracy to report:

Do I report the

  1. Accuracy on all data seen so far, calculated through just one call to accuracy with outputs and targets from all data seen so far?

or

  1. The average accuracy over individual tasks, calculated by averaging across each task (respective classes) accuracy over the stream seen so far?

thank you !

arthurdouillard commented 2 years ago

In my paper PODNet (eccv2020), but also other authors (like DER cvpr2021) only reports the avg (2). Although, in continual segmentation (https://github.com/arthurdouillard/CVPR2021_PLOP) we like to report both the avg (2) and the accuracy (1).

In my next paper, soon on arxiv, I choose to report both. I think more metrics is rarely bad.

The avg is important (you cannot be good only at the final step, you need to always be good). But the final accuracy is also important, and the difference between algorithms is bigger with this metric.

ashok-arjun commented 2 years ago

Got it, thanks a lot for the reply!