Open pp1016 opened 2 years ago
Btw, is the linear evaluation support for CIFAR dataset?
Hi,
1) yes it was a typo, I have just fixed it. 2) Ok, can you provide the script you used and the full output error?
Offline linear evaluation does not improve on CIFAR so we just use online linear eval.
Thanks for your reply!
Basically, when I run the example script with Fine-tune Barlow on CIFAR: DATA_DIR=/path/to/data/dir/ CUDA_VISIBLE_DEVICES=0 python job_launcher.py --script bash_files/continual/cifar/barlow.sh. It shows up an error "File "cassle/methods/base.py", line 432, in training_step self.knn( and File "cassle/utils/knn.py", line 89, in compute test_features = torch.cat(self.test_features) RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat. This usually means that this function requires a non-empty list of Tensors." It seems that in base.py line 432, when we call self.knn(), only train_features/targets are specified and test_features/targets are missing, this cause the raised error. When I manually set self.disable_knn_eval = True, the code can run without error.
Thanks for your help!
I encountered the same error "File "cassle/methods/base.py", line 432, in training_step self.knn( and File "cassle/utils/knn.py", line 89, in compute test_features = torch.cat(self.test_features)...", And I did the same thing with @pp1016 to set --disable_knn_eval in bash file "barlow.sh",and another error was encountered:
Traceback (most recent call last):
File "main_pretrain.py", line 224, in training_step
is {loss}.")
ValueError: The loss returned in training_step
is nan.
Is there any way to solve it? Thanks for your help! @DonkeyShot21
I was able to solve the issue.
In https://github.com/DonkeyShot21/cassle/blob/main/cassle/methods/base.py#L432 and https://github.com/DonkeyShot21/cassle/blob/main/cassle/methods/base.py#L464 there should be self.knn.update
instead of self.knn
. Calling metric()
does metric.update()
and then metric.compute()
but in training_step
and validation_step
we only want to call update()
as not all the data has been passed to the metric yet.
However, when I run the code with such fix the results are not as expected: for finetuning of BarlowTwins I get ~55% accuracy for online linear probe but only ~35% of knn accuraccy. According to the paper the difference should not be so high (ImageNet100 in Tab. 4 vs backbone eval in Tab. C).
Does anyone have a clue why the difference is so high? @DonkeyShot21 @vturrisi ?
Hi! I am not sure I understand the issue. Table C reports results using offline k-nn, while I think you are running online k-nn, am I right? Table 4 is linear evaluation, that is well known to be better than k-nn, especially with CNNs.
There are two main differences: (a) the features are collected with different parameter configs, (b) as you said, augmentations. While (a) should not be too bad at the end of training where the lr is very low, (b) is still a big issue. Normally people use no augmentations for knn afaik. Also remember that, especially on CIFAR, these augmentations are very strong and they corrupt images completely, in some cases they crop up to 3px x 3px. We never changed the hyperparams, so those that you linked should be correct.
Hi, I found this work very interesting and plan to work on similar topics. However I encounter some issues: (1) For the fine-tuning example with Barlow Twins and CIFAR-100, should it be barlow.sh instead of barlow_distill.sh? Otherwise, we need to provide the pretrained model in order to successfully run the code. (2) If I enable the the KNN online evaluation by setting disable_knn_eval = False, there was an issue showing empty test feature and expect argument in base.py line 432. I saw the previous closed issue saying the similar thing but it still appears even if I set a meaningful online_eval_batch_size = 256. Thanks for your help!