Add --backend support to bench command and default to custom image - Githubissues

triton-inference-server / triton_cli

Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inference Server.

48 stars 2 forks source link

Add --backend support to bench command and default to custom image #27

Closed rmccorm4 closed 8 months ago

rmccorm4 commented 8 months ago

Docker Changes

Use subprocess for docker pull - adds simpler progress for pulling images as expected:

Add docker build support, along with a custom Dockerfile with both vLLM and TRT-LLM dependencies that will be built/used by default.
- TODO to consider - Support launching a container to do TRT LLM engine build, in addition to existing logic for server start?
Change default server image to the custom Dockerfile based image

"bench" command changes

Add and passthrough --backend. ex: triton bench -m gpt2 --backend tensorrtllm
- Also add "vllm" to the list of "supported backends"
Add progress bars to "warming up..." and "profiling..." in profiler
Fix "trtllm" -> "tensorrtllm" backend check in profiler

TRT-LLM changes

Don't use mpirun for world_size==1
- It appears to function correctly without it, but mpirun seems to hide/obscure errors when they occured
- We can undo this later if we come to realize mpirun is necessary for world size of 1, but it doesn't currently appear so