Update metric name from input/output tokens to input/output sequence lengths

triton-inference-server / client

Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.

BSD 3-Clause "New" or "Revised" License

517 stars 224 forks source link

Update metric name from input/output tokens to input/output sequence lengths #701

Closed nv-hwoo closed 3 weeks ago

nv-hwoo commented 3 weeks ago

Change the metric names to align across NVIDIA teams. This PR only includes changing the input and output token name to ISL/OSL:

Input token --> Input sequence length (ISL)
Output token --> Output sequence length (OSL)

The PR changes include:

Update LLM metrics in llm_metrics.py
Update console display
Update README
Update default plot names
Update few other files in the codebase that uses input/output token metric name.

This PR does NOT include changing CLI option names (such as --synthetic-input-token-mean) as that is a major breaking changes to our tool.

Before:

After:

nv-hwoo commented 3 weeks ago

CI ref: ~~15752543~~ 15807804

debermudez commented 3 weeks ago

Is this PR going to update the exporter's too? I see the tests were updated and they must be passing now but that makes me think the exporter tests are not doing what they are supposed to.

nv-hwoo commented 3 weeks ago

@debermudez Where in the exporter do you think needs to change? Atm I don't see anywhere in the exporter code that requires a change. All the changes in the exporter tests are either expected results or inputs to the exporter.

debermudez commented 3 weeks ago

@debermudez Where in the exporter do you think needs to change? Atm I don't see anywhere in the exporter code that requires a change. All the changes in the exporter tests are either expected results or inputs to the exporter.

Discussed offline. Design was more stable than i realized and @nv-hwoo did a great job updating the rest.