Contributing new distillation related trainers

[YihanCao123]

Method description

Hi team, I have implemented some distillation based trainers, and would like contribute them to trl. Do you accept contributions on this or probably this is something already in progress? I see GKD has already been added into the trainer list, but some of the more basic distillation methods, as well as other knowledge distillation techniques, haven’t been included yet. I’d be happy to help expand if there’s interest. Thanks!

Open source status

• [X] The method implementation is available
• [ ] The model weights are available
• [ ] The training datasets are available

Provide useful links for the implementation

No response

[kashif]

yes would welcome distillation trainers!