icarl results?

[qsunyuan]

Nice work.

In Podnet, the experiments protocol is first trained on half classes (50 for CIFAR100), also knowns as CIFAR100-B50.

I also achieve similar results about 53.78 or 58.08 in ur paper (table 1).

However, when i tried CIFAR100-B0, which is split the whole dataset class. (original icarl experiments protocol), I cannot achieve the results. (eg, 0.64 for 10 tasks, 10 classes per task)

How's ur results?

Could u pls give some insghts?

Thx.

[qsunyuan]

(eg, average acc, 0.64, CIFAR100 for 10 tasks, 10 classes per task)

[arthurdouillard]

What did you get?

I've launch an exp on my side to see the results. I'll keep you posted.

Since you use the word "CIFAR100-B0" I assume you get that wording from the DER paper, have you tried their iCaRL version? They got around 65.27 and their code is a slightly modified version of mine so you won't be lost.

[qsunyuan]

Currently, I got a very poor results of iCaRL (CIFAR100-B0). I am rechecking the code for errors. But I achieve the similar results of ur paper results CIFAR100-B50. I'm so confused.

Yes, DER is a good reference, in Table 1, They got around 65.27 of "CIFAR100-B0" with resnet18 with more channels.(not resnet32)

In their DER paper appendix, table 7, or original iCarl paper, the results is about 64.

[arthurdouillard]

And what about their icarl code? Have you tried it? :)

Here are my icarl results I've just got, with my master branch:

Current acc: {'total': 0.45, '00-09': 0.439, '10-19': 0.387, '20-29': 0.309, '30-39': 0.321, '40-49': 0.437, '50-59': 0.542, '60-69': 0.418, '70-79': 0.517, '80-89': 0.565, '90-99': 0.568}.
avg: 63.73, last: 45.0, forgetting: 17.41

So super close of the 0.64 results no?

My command was:

python -minclearn --autolabel --options options_dev/data/cifar100_1order.yaml options/icarl/icarl_cifar100.yaml --initial-increment 10 --increment 10 --device 2

With the file options_dev/data/cifar100_1order.yaml being only the first order:

dataset: cifar100
order:
  - [87, 0, 52, 58, 44, 91, 68, 97, 51, 15, 94, 92, 10, 72, 49, 78, 61, 14, 8, 86, 84, 96, 18, 24, 32, 45, 88, 11, 4, 67, 69, 66, 77, 47, 79, 93, 29, 50, 57, 83, 17, 81, 41, 12, 37, 59, 25, 20, 80, 73, 1, 28, 6, 46, 62, 82, 53, 9, 31, 75, 38, 63, 33, 74, 27, 22, 36, 3, 16, 21, 60, 19, 70, 90, 89, 43, 5, 42, 65, 76, 40, 30, 23, 85, 2, 95, 56, 48, 71, 64, 98, 13, 99, 7, 34, 55, 54, 26, 35, 39]

seed: [1]

Note that my lib versions are:

torch                              1.2.0
torchvision                        0.4.0

I know that some models were worse once I've upgraded torch to a version above 1.2.0.

[arthurdouillard]

I'm closing the issue because it seems solved. Re-open if you have any related issues.

[qsunyuan]

Thanks for the detailed explanation, I finally achieved the result and I I still need to study some code details.

[qsunyuan]

Sorry to bother you again, I recently found that when in the first task, all the feature is 0 from the 17th dimension, can you give some insight?

https://github.com/arthurdouillard/incremental_learning.pytorch/blob/0d25c2e12bde4a4a25f81d5e316751c90e6f789b/inclearn/models/icarl.py#L302

outputs["features"][:,17:]
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0',
       grad_fn=<SliceBackward>)

[qsunyuan]

@arthurdouillard

[qsunyuan]

Thx in advance.

[arthurdouillard]

Humm... that's super weird, to be perfectly zero it sounds like a ReLU.

The first 16 dimensions are not zero tho?