search

SaynaEbrahimi / UCB

Original PyTorch implementation of Uncertainty-guided Continual Learning with Bayesian Neural Networks, ICLR 2020
https://openreview.net/pdf?id=HklUCCVKDB
MIT License
73 stars 11 forks source link

Single-head Implementation #9

Open ardywibowo opened 3 years ago

ardywibowo commented 3 years ago

Hello, I was trying to find out the code to reproduce the results from Table 3, specifically the implementation for single-head results, but it doesn't look like it is available in this repository. Is this correct? Can you point me towards how I can run the single-head method or where/how it was implemented?

ardywibowo commented 3 years ago

I ran the implementation as is and it seems the results in the paper are not reproducible?

skeskinen commented 10 months ago

I'm a couple years late to the party, but it still seems like the code for single-head (table 3) is not available.

I tried to implement it myself based on this code base, and it only gets ~40% accuracy on 5 split MNIST instead of the claimed 98.7%.

I think they must have done something like "labels trick", i.e. calculating the loss separately for each task during training. So I believe that the task description in the paper is misleading and they did not validate UCB in a proper class-IL setting. In the single-head section of the paper it says twice that "the task information is not given at test time," which could be taken as an indication that task information was available during training.

They also said it was "interesting" that the performance didn't drop going from multi-head to single-head, but didn't give (imo) a proper explanation or code. I find this rather annoying since now if you want to cite this paper in class-IL work, you need to first explain why the new results are 3000% worse than the original ones.

Of course we may never know for sure unless the authors comment or somebody figures out how to reproduce the results.