Open JoeyTPChou opened 1 year ago
All 3 scripts under bloom-inference-scripts incorrectly benchmark the t_generate_span time. The t_generate_span is got from the first generate() call at here https://github.com/huggingface/transformers-bloom-inference/blob/main/bloom-inference-scripts/bloom-ds-inference.py#L257 instead of in the benchmark cycle.
bloom-inference-scripts
t_generate_span
generate()
All 3 scripts under
bloom-inference-scriptsincorrectly benchmark thet_generate_spantime. Thet_generate_spanis got from the firstgenerate()call at here https://github.com/huggingface/transformers-bloom-inference/blob/main/bloom-inference-scripts/bloom-ds-inference.py#L257 instead of in the benchmark cycle.