Open Yeom opened 1 year ago
i meet a similar problem. if ft server encouters stop token during generating, but the already generate tokens' length shorter than the max_new_tokens, the ft server will continue reply the same result, but don't stop the streaming.
client.stop_stream() is called, but it will block until the result's lenth equal the max_new_tokens.
is there any way to get out?
Description
Reproduced Steps