llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.
Apache License 2.0
69
stars
17
forks
source link
Generate incorrect text format when use pipeline defaulttransformers #53
Set pipeline: defaulttransformers and prompt_format: "'role': 'user', 'content': {instruction}" in yaml, and seems there is text format issue in generated_text as following.
Set
pipeline: defaulttransformers
andprompt_format: "'role': 'user', 'content': {instruction}"
in yaml, and seems there is text format issue in generated_text as following.[{"generated_text":"'role': 'user', 'content': hello nihao\n{'role': 'user', 'content': '你好'}","num_input_tokens":2,"num_input_tokens_batch":2,"num_generated_tokens":26,"num_generated_tokens_batch":26,"preprocessing_time":0.007688470010180026,"generation_time":7.110702240024693,"postprocessing_time":0.0007505400571972132,"generation_time_per_token":0.2539536514294533,"generation_time_per_token_batch":0.2539536514294533,"num_total_tokens":28,"num_total_tokens_batch":28,"total_time":7.1191412500920705,"total_time_per_token":0.2542550446461454,"total_time_per_token_batch":0.2542550446461454}]