samsucik
commented
3 years ago Hi Hady,
First, sorry for the late reply.
In general, you can find a lot more details about my work in the actual thesis report https://github.com/samsucik/knowledge-distil-bert/blob/master/latex/s1513472-minf2.pdf (and, of course, all the code is in that repo as well).
I was using BERT_base, fine-tuning it on downstream tasks, and then distilling the fine-tuned model into student models. To your question (whether to distil the pre-trained or the fine-tuned model), the answer really depends on your situation. Have a look at section 2.3.2 in the report where I mention some advantages and disadvantages of each option :-)
As for teacher assistants, the intermediate model (the assistant) doesn't have to have the same architecture as the teacher. It's the same as with normal knowledge distillation, where the student can have a different architecture from the teacher. However, in some cases there can be advantages of having the same architecture: Especially if you're trying in your distillation to match not just the teacher's logits, but also its internal representations. Indeed, an LSTM student/assistant cannot learn well the internal representations encoded in a transformer-based teacher's self-attention heads, because there is no such component in an LSTM.
Let me know if you've got more specific questions, I'll try to reply faster this time.
Best of luck!
Sam
On Tue, 23 Mar 2021 at 12:26, Hady walied @.***> wrote:
Hello, I'm Hady an ECE student at cairo university school of engineering, I've been working on a distilled version of a text summarization model called pegasus, I found your L3-AI talk on YouTube and I'll be really thankful if you answer some of my questions.
I was wondering if the technique you used to distill bert was applied on a downstreamed version of the model or on the BERT_large version, meaning that if I have a model that is pre trained on the c4 dataset for example, and then downstreamed on the summarization task by training it on gigaword dataset, when I try to distill the model should I start with the pretrained version or the downstream version?
Also, I wanted to know if I used a Teacher Assistant technique. Should it have the same architecture as the teacher model or can I alter the architecture?
I look forward to your kind reply.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/samsucik/knowledge-distil-bert/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABY7YZQMQBKKQIOZEA5Y2L3TFCCI5ANCNFSM4ZVCMFWQ .
hadywalied
Hello, I'm Hady an ECE student at cairo university school of engineering, I've been working on a distilled version of a text summarization model called pegasus, I found your L3-AI talk on YouTube and I'll be really thankful if you answer some of my questions.
I was wondering if the technique you used to distill bert was applied on a downstreamed version of the model or on the BERT_large version, meaning that if I have a model that is pre trained on the c4 dataset for example, and then downstreamed on the summarization task by training it on gigaword dataset, when I try to distill the model should I start with the pretrained version or the downstream version?
Also, I wanted to know if I used a Teacher Assistant technique. Should it have the same architecture as the teacher model or can I alter the architecture?
I look forward to your kind reply.