Support for Llama 3.1 - Githubissues

npuichigo / openai_trtllm

OpenAI compatible API for TensorRT LLM triton backend

MIT License

176 stars 27 forks source link

Support for Llama 3.1 #52

Open datdo-msft opened 3 months ago

datdo-msft commented 3 months ago

Is it possible to get support for the new Llama 3.1 models? It seems they are using a new chat template, at least from what I saw when I looked at tokenizer_config.json for 405B-FP8-Instruct on HuggingFace.