Closed kiucho closed 6 months ago
I think it should work for pruning fine-tuned models! The key is to add back the fine-tuning data at some point, either at the first stage (pruning) or second stage (continued pre-training).
Thanks for reply! I'll try to make some progress.
Thanks for sharing your great research.
I was wondering if pruning could be applied to an already fine-tuned model, so I'd like to ask for your advice. If so, would it be possible to have continued pre-training?