search

triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend
Apache License 2.0
588 stars 81 forks source link

the tensorrtllm backends and onnxruntime backends #401

Closed tricky61 closed 2 months ago

tricky61 commented 2 months ago

I have install this image: tritonserver-23.10-trtllm-python-py3 but the backends only has tensorrtllm and python, Can I use previous backends like: onnxruntime、pytorch、tensorrt...? how to add them

byshiue commented 2 months ago

For specific backend docker image, it only installs the corresponding backend to save the memory. If you want to install several backends in one docker image, you could refer the document of triton backend about building docker image. Hope this document https://github.com/triton-inference-server/backend?tab=readme-ov-file#can-i-add-or-remove-a-backend-to-an-existing-triton-installation is helpful.

tricky61 commented 2 months ago

For specific backend docker image, it only installs the corresponding backend to save the memory. If you want to install several backends in one docker image, you could refer the document of triton backend about building docker image. Hope this document https://github.com/triton-inference-server/backend?tab=readme-ov-file#can-i-add-or-remove-a-backend-to-an-existing-triton-installation is helpful.

Thanks for your reply