Early Exit during inference

HengJayWang / Llama-2-7b-Finetuned-Early-Exit

Fine tuning the Llama-2-7b model for Early Exit inference.

0 stars 0 forks source link

Early Exit during inference #1

Open xlim1996 opened 4 months ago

xlim1996 commented 4 months ago

Hi,

Thank you for your excellent work. I noticed that while you trained the model with early exit at fixed intermediate layers, it seems that this capability has not been utilized during inference, based on my understanding. Could you provide the code for using early-exit during inference?

Best regards, Xlim

HengJayWang commented 4 months ago

Hi Xlim @xlim1996,

I did not write the inference code for early exit, I just evaluate full fine-tune model with EleutherAI framework https://github.com/EleutherAI/lm-evaluation-harness

But you can reference my partner repo to edit some custom early exit condition with HuggingFace transformers library. https://github.com/kevin1010607/LLM_convex

Thanks~🥰