Open AllentDan opened 7 months ago
TurbomindEngineConfig
prompts
to input_ids
?
- It's better to deprecate model_name instead of removing it directly. PR Remove model name when loading hf model #1022 is working on it for
TurbomindEngineConfig
- For the pipeline case, are you suggesting changing
prompts
toinput_ids
?
PytorchEngineConfig
then. model_name
still exists in ChatTemplateConfig
whic is not gonna be deprecated.chat_template
to tokenizer in pipeline.Should we consider making the chat template or model_name a required argument? The matching strategy is quite prone to failure or mismatching to the wrong template.
I prefer an accurate matching like FastChat.
The /v1/completions
interface does not consider any dialogue template issues and only accepts an input string, which is directly processed by the tokenizer and then inferred. Therefore, only /v1/chat/completions
and /v1/chat/interactive
are discussed:
model_name
, no jinja-template
:
/v1/chat/completions
prioritizes reading the Jinja template from tokenizer_config.json
, and if not available, uses the one defined by LMDeploy./v1/chat/interactive
uses the one defined by LMDeploy.model_name
provided, no jinja-template
:
model_name
in LMDeploy format.model_name
provided, jinja-template
provided:
/v1/chat/completions
uses the jinja-template
./v1/chat/interactive
is not guaranteed, and it usually responds to the request as text completion.model_name
provided, jinja-template
provided:
model_name
finds a dialogue template, all interfaces use the LMDeploy dialogue template.model_name
does not find a dialogue template, an error is reported directly, and the program cannot be started.
Motivation
model_name
from EngineConfig to avoid redundant specification.Major features
Tokenizer
class supports Transformers’ Jinja dialogue templates.model.get_prompt
will be moved toTokenizer.apply_chat_template
model_name
is removed fromTurbomindEngineConfig
andPytorchEngineConfig
.How to use
For api_server, to use an extra template, commands could be:
For APIs like
pipeline
, we are going to provide documents to show how to add a chat template in python or Jinjia. The codes will be: