search

SafeAILab / EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)
https://arxiv.org/pdf/2406.16858
Apache License 2.0
826 stars 81 forks source link

Eagle head accuracy #17

Closed zankner closed 10 months ago

zankner commented 11 months ago

Hello, I was just wondering what your top-1 accuracy is for eagle heads.

Liyuhui-12 commented 11 months ago

Top-1 accuracy is just under 0.8. We will provide detailed experimental results and analysis in the paper.

zankner commented 11 months ago

Any chance you would be able to share the plot of accuracy throughout training? I'm trying to replicate the results in my own codebase and getting performance around 0.4 so assuming I'm doing something wrong.

Liyuhui-12 commented 11 months ago

Of course, you can see the plot for Vicuna 13B in https://api.wandb.ai/links/yuhui-li/kgzd2kc8.

je1lee commented 10 months ago

@Liyuhui-12

With my understanding, "test/top_1_acc" is top_1 accuracy of EAGLE head accuracy on next token prediction and "test/0_acc" also designate same accuracy. But there's difference in metrics. I guess my understanding is wrong could you tell me the difference between "test/top_1_acc", "test/0_acc"

Liyuhui-12 commented 10 months ago

They are roughly the same, the difference is that test/0_acc does not calculate the tokens in the dataset that are inconsistent with the original LLM predictions

qspang commented 10 months ago

can you provide code for testing head_accuracy like the medusa project?https://github.com/FasterDecoding/Medusa/blob/v1.0-prerelease/medusa/eval/heads_accuracy.py