Open yiguanxian opened 7 months ago
复现代码块: 模型:baichuan2-13b-chat import pyfastllm import time
model = pyfastllm.create_llm(model_path) handle = model .launch_response(input_ids, fastllm.GenerationConfig()) continue_token = True ret_str = "" start_time = time.time() while continue_token: resp_token = self.model.fetch_response(handle) end_time = time.time() print("time:", end_time - start_time) 结果1: time: 0.54363 time: 0.74165 time: 1.14061 time: 2.4361 结果2: time: 2.34163 time: 2.35263 time: 2.35312 time: 2.35681 结果分析:它们输入prompt都是一样的,多次运行,结果因为大模型的采样选择所以可能不一样。可以看到首个token出来的时间不稳定,有时候会慢一点,有时候会快一点。当首个token比较慢出来时,表现出来就是后面的所有token会一次性吐出来,这样给人的感觉很不“流式”。
复现代码块: 模型:baichuan2-13b-chat import pyfastllm import time
model = pyfastllm.create_llm(model_path) handle = model .launch_response(input_ids, fastllm.GenerationConfig()) continue_token = True ret_str = "" start_time = time.time() while continue_token: resp_token = self.model.fetch_response(handle) end_time = time.time() print("time:", end_time - start_time) 结果1: time: 0.54363 time: 0.74165 time: 1.14061 time: 2.4361 结果2: time: 2.34163 time: 2.35263 time: 2.35312 time: 2.35681 结果分析:它们输入prompt都是一样的,多次运行,结果因为大模型的采样选择所以可能不一样。可以看到首个token出来的时间不稳定,有时候会慢一点,有时候会快一点。当首个token比较慢出来时,表现出来就是后面的所有token会一次性吐出来,这样给人的感觉很不“流式”。