Closed msdmkats closed 3 months ago
We will need to do a full slicing and finetuning run to verify that the dependencies (datasets, transformers, peft) don't lead to issues. I've spent a while debugging why finetuning loss was just
Nan
, turned out peft versions are not backwards compatible. Pinning to peft=0.6.0 solved the issue. This may be due to the interface changing, so I suggest we do a full check before updating dependencies (not just the tests).
Peft 0.9.0 indeed caused problems. I rolled it back to 0.6.0, but also made optional as it apparently is also not needed for the actual slicing. Other than that, logs for finetuning with parameters from readme for phi-2 sliced with sparsity 0.25 using alpaca for calibration end with
{'loss': 1.384, 'grad_norm': 39.89509582519531, 'learning_rate': 0.00012, 'epoch': 0.17}
{'eval_loss': 1.1535898447036743, 'eval_runtime': 16.2278, 'eval_samples_per_second': 7.888, 'eval_steps_per_second': 0.986, 'epoch': 0.17}
And lm_eval's score on piqa task for that model is 0.745 both before and after finetuning.
Update some dependencies, and make dependencies that are not required for slicing optional.