In the EEIL paper, distillation loss is applied to classification layers corresponding to previous classes, and the balanced finetuning stage adds temporary distillation loss for classification layers for new classes.
But my question is, in the balanced finetuning stage, I can't see what the temporary distillation loss for classification layer for the new class is. Also, looking at the loss function part of the EEIL code, it appears that during the balanced fine-tuning stage, the fc layer is computed to correspond to task t-1 for the distillation. On the other hand, the distillation is applied to the fc layer up to task-2 before entering the balanced fine-tuning stage. If we calculate like this, doesn't distillation in the unbalanced training stage apply to the classifier corresponding to task t-1?
I'm asking because I get confused even if I look at EEIL paper and code several times.
Thank you.
In the EEIL paper, distillation loss is applied to classification layers corresponding to previous classes, and the balanced finetuning stage adds temporary distillation loss for classification layers for new classes. But my question is, in the balanced finetuning stage, I can't see what the temporary distillation loss for classification layer for the new class is. Also, looking at the loss function part of the EEIL code, it appears that during the balanced fine-tuning stage, the fc layer is computed to correspond to task t-1 for the distillation. On the other hand, the distillation is applied to the fc layer up to task-2 before entering the balanced fine-tuning stage. If we calculate like this, doesn't distillation in the unbalanced training stage apply to the classifier corresponding to task t-1?
I'm asking because I get confused even if I look at EEIL paper and code several times. Thank you.