Support P50, P90, P99 for next token latency

opea-project / GenAIEval

Evaluation, benchmark, and scorecard, targeting for performance on throughput and latency, accuracy on popular evaluation harness, safety, and hallucination

Apache License 2.0

18 stars 29 forks source link

Support P50, P90, P99 for next token latency #93

Closed lvliang-intel closed 2 weeks ago

lvliang-intel commented 2 weeks ago

Description

Support P50, P90, P99 for next token latency

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would break existing design and interface)

Dependencies

None

Tests

Local test.