search

CentML / flexible-inference-bench

A modular, extensible LLM inference benchmarking framework that supports multiple benchmarking frameworks and paradigms.
Apache License 2.0
5 stars 0 forks source link

[ADDED ARGS] added new args: stream, cookies #44

Closed johncalesp closed 5 months ago

johncalesp commented 5 months ago
  1. Added cookie option for requests
  2. Added stream as argument because sometimes we may need to test non-stream requests
  3. Changed performance.py to accomodate for stream and non-stream output files.
andoorve commented 5 months ago

It seems like vLLM folks had the exact same idea recently! Apparently ITLs aren't interchangeable with TPOTs and we should use both. Sorry for the confusion!

https://github.com/vllm-project/vllm/compare/d5b1eb081e19...02cc3b51a7f2

johncalesp commented 5 months ago

added the necessary changes for itls