search

SafeAILab / EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)
https://arxiv.org/pdf/2406.16858
Apache License 2.0
780 stars 79 forks source link

Can EAGLE actually improve throughput? #36

Closed chenzhengda closed 7 months ago

chenzhengda commented 7 months ago

It seems that it always needs a complete forward of tree candidates for verify, which appears to increase the overall computational flops. For example, for "mc_sim_7b_63," each iteration requires the computation of 26 candidate tokens, but only two tokens can be accepted.

hongyanz commented 7 months ago

EAGLE alone achieves 2x throughput. See Table 7 in the paper: https://arxiv.org/abs/2401.15077