search

triton-inference-server / triton_cli

Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inference Server.
48 stars 2 forks source link

Enable CLI Concurrent Testing #41

Closed fpetrini15 closed 7 months ago

fpetrini15 commented 7 months ago

Primary Goal: Adjust the testing structure to accept models passed in via the environment to enable testing with multiple models to run concurrently.

Secondary Goal: Add opt125m to set of CLI test models.

Note: psutil and pynvml were removed from the requirements list. psutil was removed because it is installed as part of the CI workflow. pynvml was removed because it requires special handling from the CI side to accommodate compatibility issues on our runners. pynvml>=11.5.0 is already installed as part of installing TRT LLM v0.8.0.