Is there any way to distill translation models?

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

MIT License

30.43k stars 6.4k forks source link

Open AIikai opened 2 years ago

AIikai commented 2 years ago

Prepare to compress the translation model trained by fairseq, such as distillation and pruning. Can you give me some advice? Thank U.

robotsp commented 2 years ago

@AIikai I have a similar question. Do you have any ideas?

VarunGumma commented 2 years ago

@AIikai @robotsp I am also looking to distill and prune a few LLMs. Any leads?

HeegonJin commented 2 years ago

@AIikai @robotsp @VarunGumma I am also trying to implement KD with the modification of fairseq

VarunGumma commented 2 years ago

@robotsp @AIikai @HeegonJin please redirect here for KD in fairseq