Use a deterministic split for forget and retain sets

[bkowshik]

Before this PR

Results from Jupyter notebook in the main branch. For each of the datasets, I ran the 3 models and calculated accuracy. Results of the fine-tuned model vary from run to run, this is expected given we are fine-tuning only for 5 epochs.

Dataset	Model	Fine-tuned model	Re-trained model
Train	99.5	84.6	98.4
Retain	99.5	85.2	99.5
Forget	99.3	79.6	88.2 ✅
Test	88.3	78.3	88.3
Valid	88.9	78.4	87.6

With this PR

Took the notebook from this branch and ran the same procedure again and below are the results.

Dataset	Model	Fine-tuned model	Re-trained model
Train	99.5	97.0	98.4
Retain	99.5	98.4	99.5
Forget	99.3	85.0	88.2 ✅
Test	88.3	83.0	88.0
Valid	88.9	83.0	88.0

Notes

Clearly, the results I was getting from a modified version of the starter kit was different which made me create the ticket. But, I am not able to replicate those results from the starter kit which very likely hints at an issue with my changes. 😞

Below are the snippets I appened at the end of the notebook to get the results:

print(f"Train set accuracy: {100.0 * accuracy(model, train_loader):0.1f}%")
print(f"Retain set accuracy: {100.0 * accuracy(model, retain_loader):0.1f}%")
print(f"Forget set accuracy: {100.0 * accuracy(model, forget_loader):0.1f}%")
print(f"Test set accuracy: {100.0 * accuracy(model, test_loader):0.1f}%")
print(f"Valid set accuracy: {100.0 * accuracy(model, val_loader):0.1f}%")

print(f"Train set accuracy: {100.0 * accuracy(ft_model, train_loader):0.1f}%")
print(f"Retain set accuracy: {100.0 * accuracy(ft_model, retain_loader):0.1f}%")
print(f"Forget set accuracy: {100.0 * accuracy(ft_model, forget_loader):0.1f}%")
print(f"Test set accuracy: {100.0 * accuracy(ft_model, test_loader):0.1f}%")
print(f"Valid set accuracy: {100.0 * accuracy(ft_model, val_loader):0.1f}%")

print(f"Train set accuracy: {100.0 * accuracy(rt_model, train_loader):0.1f}%")
print(f"Retain set accuracy: {100.0 * accuracy(rt_model, retain_loader):0.1f}%")
print(f"Forget set accuracy: {100.0 * accuracy(rt_model, forget_loader):0.1f}%")
print(f"Test set accuracy: {100.0 * accuracy(rt_model, test_loader):0.1f}%")
print(f"Valid set accuracy: {100.0 * accuracy(rt_model, val_loader):0.1f}%")

[seifachour1]

Certainly the fine tuned model training accuracy won't surpass logically in general the retrained one especially when the loss is too close to zéro and we don't have to find another global minimum, because basically the highest level of forgetting something is to not know it even from the beginning so it's normal that the accuracy of the retrained model on forget set can't get better than 88,2%. I explained that in my scientific article indeed.

[fabianp]

thanks @bkowshik for the comment. Even if you could not fix it in your end, I believe a deterministic split would avoid many potential issues. @eleniTriantafillou : are you OK with merging this?

[eleniTriantafillou]

yes, I agree with you Fabian! thanks for making the change!