search

SafeAILab / EAGLE

Official Implementation of EAGLE
https://arxiv.org/pdf/2406.16858
Apache License 2.0
622 stars 59 forks source link

Is there scripts to calculate the overall acceptance rate? #62

Open sugar1946 opened 2 months ago

sugar1946 commented 2 months ago

I am trying to calculate the acceptance rate for evaluation. I ran:

python -m eagle.evaluation.gen_ea_alpha_vicuna --ea-model-path ~/models/EAGLE-Vicuna-7B-v1.3/ --base-model-path ~/models/vicuna-7b-v1.3/ --model-id ess-vicuna-7b-fp16-alpha --temperature 1

However, this seems to only generates an output decoded file with alpha per question? How do I get the overall alpha rate as in the paper?

Liyuhui-12 commented 2 months ago

After obtaining the result file, you can run the eagle/evaluation/alpha.py file to get the acceptance rate.