Verify unlearning with opt1.3B but different retain set (Squad vs TruthfulQA)

[Willmish]

• [ ] Since Script changed a lot recently, repeat experiment both for:
  • [ ] TruthfulQA retain set
  • [ ] Squad retain set
• [ ] compare results

Hypothesis:

1. Similar results/curve shape for task-based evaluation (over continuous checkpoints/sequential)
2. Responses on average longer for unlearned models with Squad retain set (since squad has longer data)

[Willmish]

2024-09-17:

• Compare task performance between truthfulQA and Squad