search

princeton-nlp / CoFiPruning

[ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408
MIT License
188 stars 32 forks source link

Should I get a finetuned teacher model by setting `pruning_type=None` #15

Closed horizon86 closed 2 years ago

horizon86 commented 2 years ago

Should I get a teacher model by finetune a bert-base-uncased model by setting pruning_type=None, pretrained_pruned_model=None ,remove do_distill and do_layer_distill? then I use the finetuned model as distillation_path(teacher model) to get a pruned model?

or Should I fine tune the model myself to get the teacher model?

In the code pruning_ type=none is used to fine tune after pruning.

Thanks!

xiamengzhou commented 2 years ago

Hi,

Thanks for using our repo! Yes, your description is correct, but you can also use your own pipeline to get the teacher models. Let me know if you encounter any issues :)

xiamengzhou commented 2 years ago

Hi,

I am closing this issue. Feel free to reopen it if you have more questions :)