Closed DeepTecher closed 3 months ago
it is correct here. There are two possible reasons to finish the task: 1. the generated token_id is eos_token_id; 2. generated token_id len exceed max_new_tokens. we always add current token_id to output, and finish task based on finish_reason.
however, when we run on A100, it will raise an error if we do not change to the following modification. Please confirm it on chatglm case.
however, when we run on A100, it will raise an error if we do not change to the following modification. Please confirm it on chatglm case.
chatglm is out-of-date, and it will be removed later. There is a update today, you can try chatglm2 instead.
https://github.com/bytedance/ByteMLPerf/blob/c1d1835738cc17d974197f809fa503ec84e32d4d/byte_infer_perf/llm_perf/backends/GPU/gpu_scheduler.py#L60
Hi, We followed your script to test and found that there is a problem here, which needs to be modified