search

XiaoBin1992 / clover

Official Implementation of Clover-1 and Clover-2
Apache License 2.0
4 stars 1 forks source link

Knowledge distillation #4

Open fousdfrf opened 3 months ago

fousdfrf commented 3 months ago

I've noticed that this knowledge distillation is somewhat similar to what is mentioned in EAGLE, and it has proven to be very effective. I would like to know if you have tried the knowledge distillation methods mentioned in the paper 'DISTILLSPEC: Improving Speculative Decoding via Knowledge Distillation', such as RKL, TVD, etc.

XiaoBin1992 commented 3 months ago

Improving Speculative Decoding via Knowledge Distillation We haven't noticed the article you mentioned, but it seems that the basic idea is similar.