search

mmasana / FACIL

Framework for Analysis of Class-Incremental Learning with 12 state-of-the-art methods and 3 baselines.
https://arxiv.org/pdf/2010.15277.pdf
MIT License
512 stars 98 forks source link

The meaning of TAw & TAg #15

Closed Lllydialee closed 2 years ago

Lllydialee commented 2 years ago

Thank you very much for providing such an amazing continual learning framework!

I am a little confused about the evaluation metric: TAw acc & TAg acc, could you please explain their meaning respectively? Thank you very much in advance!!

1

mmasana commented 2 years ago

Hi @Lllydialee, TAw means task aware, and TAg means task agnostic. The first one uses the task-ID during evaluation, meaning that you only have to predict from the classes within the task. This is the case for task-incremental learning. For the second case, you do not have access to the task-ID, therefore you predict from all classes learned so far, which is the case for class-incremental learning. In the output that you show, each column represents each of the learned tasks, and each row represents at which task were those metrics calculated. That's why the top diagonal is empty, because you cannot evaluate on task 5 when you are at task 3 (task 5 has not happened yet at that point). The last column is an average over all accuracies of all tasks seen so far.

Lllydialee commented 2 years ago

Really thank you for explanation! Thanks!