triton-inference-server / client

Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
BSD 3-Clause "New" or "Revised" License
527 stars 225 forks source link

Add compare subcommand #623

Closed nv-hwoo closed 2 months ago

nv-hwoo commented 2 months ago

Add compare subcommand that allows users to generate plots to compare multiple runs.

Users can provide paths to profile export files using --files CLI option as following

genai-perf compare --files [file, ...]

which will generate a set of plots using these runs as well as pre-filled, default yaml configuration file that users can edit to tweak the plotting parameters.

Users can also directly pass the yaml config file to generate customized plots detailed in the yaml file as following

genai-perf compare --config [YAML file]

General workflow of user can be:

nv-hwoo commented 2 months ago

Would you be able to look into what the most Pythonic way of doing this is? It seems to be that having some functionality be under a subcommand and some functionality would not be the generally suggested approach. I'd also want to see if this messes up the help menu.

Great point. Yeah I am not really satisfied on how the current command/subcommand is structured, and we should definitely look into this (and I think @debermudez has already started the conversation on this). But I think that effort will require more effort than this current PR. I have TMA-1900 ticket to refactor CLI for this.