search

triton-inference-server / triton_cli

Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inference Server.
48 stars 2 forks source link

POC: Background Server #15

Closed fpetrini15 closed 10 months ago

fpetrini15 commented 10 months ago

Simple POC for launching the server in the background and running inference.

Steps:

  1. pip install triton cli
  2. triton repo add -m opt125 --source hf:facebook/opt-125m
  3. triton bench run -m opt125