Evaluate base models with beavertails.

[TheRootOf3]

Consider the following models:

• [x] LLaMa 3 8B
• [x] LLaMa 2 7B
• [x] OLMo 7B
• [x] OLMo 7B Instruction tuned
• [x] Gemma 7B
• [x] Gemma 7B instruction tuned
• [x] Aya 23 8B

[Willmish]

@Adamliu1 Llama 2, OLMo 7B

@Willmish LLama 3 8B, Gemma 7B, Aya 23 8B

Try on lab machines in half precision

[Willmish]

Model Name	Flagged/All (lower is safer)
Llama 3 8B	0.334286
Llama 3 8B repeat (same params)	0.334286
Llama 2 7B	0.311428
OLMo 7B	0.287142
OLMo 7B IT	0.372857
Gemma 7B	0.34
Gemma 7B IT	0.08
Aya 23 8B	0.227143
Opt 1.3B	0.2957142857142857

[Adamliu1]

NOTE: we should either not do IT version at all, or do it across all models.

[Willmish]

Not all have instruction tuned versions (aya is only it), but I was mostly just curious xd

[TheRootOf3]

Update as of 2024-07-01:

• Worth evaluating Gemma 2 9B. @Adamliu1 @Willmish

[Willmish]

(Fairly certain this is LR 2e-6, unlearn set: PKU-harmful, retain: squad)

Unlearned llama3 8B sequential 64; batch 1024; full precision model_name,flagged/all idx_20,0.1657142857142857

Unlearned llama3 8B sequential 16; batch 1024; full precision model_name,flagged/all idx_20,0.19

Unlearned llama3 8B sequential 4; batch 1024; full precision model_name,flagged/all idx_20,0.20714285714285716

[Willmish]

done