[Bench] Support applying chat template

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation

https://llm.mlc.ai/

Apache License 2.0

19.26k stars 1.58k forks source link

Closed MasterJH5574 closed 1 month ago

MasterJH5574 commented 1 month ago

This PR supports applying tokenizer chat template in benchmarking. It needs to be manually specified via flag --apply-chat-template.

In the following cases --apply-chat-template is not supported:

when --input-len is also specified, which means the input text needs truncation.
when the dataset is not ShareGPT.
when the tokenizer does not have chat_template defined.