alipay / PainlessInferenceAcceleration

Creative Commons Attribution 4.0 International
283 stars 18 forks source link

Do lookahead and repetition_penalty conflict? #24

Open zhanweiw opened 6 months ago

zhanweiw commented 6 months ago

After enabled repetition_penalty, will it lower lookahead's probability? If yes, any solution for avoiding the conflict?

zheyishine commented 5 months ago

It indeed may lower the speedup by about 5%-10%. A sufficient warmup could ease the negative effect.