search

npuichigo / openai_trtllm

OpenAI compatible API for TensorRT LLM triton backend
MIT License
176 stars 27 forks source link

License #16

Closed csmileyk closed 10 months ago

csmileyk commented 11 months ago

Great work on this project and very interesting. Would like to build on this and was wondering what is the license for this project?

Thanks!

npuichigo commented 11 months ago

Thanks for your interest. I have added an MIT LICENSE just now.

csmileyk commented 11 months ago

Thank you for the prompt reply. Seem to getting the following error:

{"v":0,"name":"openai_trtllm","msg":"[NON-STREAMING COMPLETIONS - EVENT] openai_trtllm::routes::completions","level":50,"hostname":"itaengai2-east-a100-vm2","pid":40013,"time":"2023-12-14T08:35:58.366371465Z","target":"openai_trtllm::routes::completions","line":123,"file":"src/routes/completions.rs","error":"AppError(error message received from triton: [request id: ] expected number of inputs between 1 and 3 but got 9 inputs for model 'model')"}

Any ideas?

npuichigo commented 11 months ago

Which model do you use? Can you check if your triton model input https://github.com/npuichigo/openai_trtllm/blob/6cf2a27a035fa7ac1b341f765114bd0aed42c844/models/baichuan/ensemble/config.pbtxt#L30 matches the router code https://github.com/npuichigo/openai_trtllm/blob/6cf2a27a035fa7ac1b341f765114bd0aed42c844/src/routes/completions.rs#L183