search

SforAiDl / KD_Lib

A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.
https://kd-lib.readthedocs.io/
MIT License
606 stars 58 forks source link

Paper: Sequence-Level Knowledge Distillation #7

Closed avishreekh closed 4 years ago

avishreekh commented 4 years ago

Sequence-Level Knowledge Distillation, Yoon Kim, Alexander M. Rush, 2016 https://arxiv.org/pdf/1606.07947.pdf

Het-Shah commented 4 years ago

This is a great paper on the NLP side of things. You can later try something along these lines for transformer-based networks.

Het-Shah commented 4 years ago

You can add your name to CONTRIBUTORS.rst. :smile:

avishreekh commented 4 years ago

This is a great paper on the NLP side of things. You can later try something along these lines for transformer-based networks.

I actually found an ICLR paper for transformers. Will definitely think of that after this.

Het-Shah commented 4 years ago

I actually found an ICLR paper for transformers. Will definitely think of that after this.

Yup I have read that paper I guess, is it TinyBERT?