[Re] Bio-inspired replay for transformers in Continual Learning

[disha101003]

Original article: https://arxiv.org/abs/2305.04769

PDF URL: https://github.com/disha101003/ReScience/blob/master/Re%5BBiRT%5D.pdf Metadata URL: Code URL: https://github.com/disha101003/ReScience

Scientific domain: Computer Vision and Pattern Recognition Programming language: Python Suggested editor:

[rougier]

Thanks for your submission and verry sorry for the long delay. We'll assign an editor soon. @gdetor @benoit-girard Can you handle this submission ?

[gdetor]

@rougier I prefer to review this work

[disha101003]

@gdetor Thank you

[rougier]

I'll edit this submission then.

@zhywan @MollyZhang @arianesasso Can you review this submission ? Bio-inspired replay for transformers in Continual Learning

[gdetor]

Hi @rougier and @disha101003 Here is my report:

Main Text

1. Try to explain any variable or parameter when you introduce them. For instance, in Equation 1, you should say that \theta are the model parameters.
2. You can drop the parenthesis when you write a function, f, or g.
3. There are minor typos, such as missing spaces (page 5 before algorithm 1).
4. In algorithm 3, there are missing indices, b[i] and l[i].
5. On page 6, there is a missing reference (Table ??).
6. On page 7, there is a missing reference (Figure ??).
7. I believe that Figures 4 (page 9) and 3 (page 10) are incorrectly numbered. Please verify in the main text that the references are correct, too.
8. In section 8.1, the first paragraph, there is a typo: a missing space between the words "experiment" and "Time."
9. The figures' fonts could be a little bit larger.
10. The table references in Section 8.4 are incorrect. The current references concern the CIFAR10 data set, not the CIFAR100 data set.
11. Does Figure 7 refer to CIFAR10 or CIFAR100?

Source Code

1. The README file needs improvements. Please add more information about the platform where the code was executed (e.g., type of CPU/GPU, GCC compiler, Pytorch, and Python version).
2. The README file's usage examples are incorrect. The user should use the command python3 train_task instead of python3 train. Please correct that.
3. The source code is not sufficiently documented. Although many comments explain the steps within methods and functions, there are no docstrings about arguments and variables.
4. It would be better if the source code complied with the PEP8.
5. It would be better if the versions of Python packages were specified in the requirements.txt file.
6. The models take some time to run; thus, it would be great if you could upload serialized versions of the pre-trained parameters of the models so we could run the test scripts without training them beforehand.
7. It is somehow redundant to print both the number of epochs and have the progress bar running globally simultaneously at the stdout. I assume the progress bar was meant to be used for each epoch, which does not seem to work.

[disha101003]

Hi, @gdetor Thank you so much for your feedback, I will address those and let you know once I am done

[disha101003]

Hi @gdetor @rougier I finished addressing all the feedback. Please let me know if anything else needs to be done or if you approve the submission.

[gdetor]

Hi @rougier, @disha101003 addressed all my comments/concerns. I have no further comments.

[disha101003]

@rougier Thank you, @gdetor any updates on the second review ?