search

alipay / PainlessInferenceAcceleration

Creative Commons Attribution 4.0 International
283 stars 18 forks source link

when the batchSzie > 1, the lookahead not work. #17

Closed yuenyu1 closed 7 months ago

yuenyu1 commented 7 months ago

I meet a bug when I run baichuan2_7b_batch_example. when the batchSzie > 1, the lookahead not work. and only one answer output.