Closed fabianp closed 1 year ago
Certainly the fine tuned model training accuracy won't surpass logically in general the retrained one especially when the loss is too close to zéro and we don't have to find another global minimum, because basically the highest level of forgetting something is to not know it even from the beginning so it's normal that the accuracy of the retrained model on forget set can't get better than 88,2%. I explained that in my scientific article indeed.
thanks @bkowshik for the comment. Even if you could not fix it in your end, I believe a deterministic split would avoid many potential issues. @eleniTriantafillou : are you OK with merging this?
yes, I agree with you Fabian! thanks for making the change!
Before this PR
Results from Jupyter notebook in the
main
branch. For each of the datasets, I ran the 3 models and calculatedaccuracy
. Results of the fine-tuned model vary from run to run, this is expected given we are fine-tuning only for 5 epochs.With this PR
Took the notebook from this branch and ran the same procedure again and below are the results.
Notes
Clearly, the results I was getting from a modified version of the starter kit was different which made me create the ticket. But, I am not able to replicate those results from the starter kit which very likely hints at an issue with my changes. 😞
Below are the snippets I appened at the end of the notebook to get the results: