Prototype tool with simple client, repo, and server features

triton-inference-server / triton_cli

Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inference Server.

48 stars 2 forks source link

Prototype tool with simple client, repo, and server features #3

Closed rmccorm4 closed 10 months ago

rmccorm4 commented 11 months ago

# Install package, also installs executable called 'triton'
pip install /path/to/triton_cli/

# Interact with repos
triton repo list
triton repo clear
triton repo add --name opt125 --hf facebook/opt-125m

# Start server (default docker)
triton server start
# or local
triton server start --mode local --repo /mnt/models

# Infer with client
triton client infer -m opt125

TODO:

Add some tests
Clean up or reduce unused code in TritonServerConfig for now