search

Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.47k stars 3.39k forks source link

An example for knowledge distillation. Especially for the load the teacher model's weights , and train the student model #17592

Open emilyemliyM opened 1 year ago

emilyemliyM commented 1 year ago

Description & Motivation

No response

Pitch

A example for knowledge distillation. Especially for the load the teacher model's weight ,and train the student model.

Now I have a trained teacher_model. When I want to train the student model, I need to load the teacher model's weights and set the teacher model to eval().

I am confused about the specific steps to do above using the Lightning. Could I get an example for it?

Alternatives

An example for knowledge distillation. Especially for the load the teacher model's weight ,and train the student model

Additional context

An example for knowledge distillation. Especially for the load the teacher model's weight ,and train the student model.

cc @borda

perceptiveshawty commented 1 year ago

I am interested in having this as a feature, I'll work on an example/tutorial and get back to you

HeegonJin commented 11 months ago

Any news?