The meaning of the CHAT option in decoding

hao-ai-lab / LookaheadDecoding

Apache License 2.0

1.04k stars 63 forks source link

Closed YingHH1 closed 7 months ago

YingHH1 commented 7 months ago

What are the differences between using CHAT=1 and CHAT=0 during the decoding process?

Many thanks!

Viol2000 commented 7 months ago

If you use CHAT=1, every generated token will be printed on your screen in real time. Otherwise, the output will not be printed in real time.