search

janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)
https://jan.ai/
GNU Affero General Public License v3.0
21.82k stars 1.25k forks source link

feat: Docker container for TensorRT-LLM environment #2862

Open iamkucuk opened 4 months ago

iamkucuk commented 4 months ago

Problem TensorRT-LLM is one of the biggest features in jan. As the docker environments usually include all kind of stuff to run the software with it's full set of features, why not include it too?

Success Criteria A container that supports TensorRT-LLM backend out-of-the-box.

sammcj commented 2 months ago

Any update on the TensorRT Jan image?