How to use llmperf to test ollama performance (TTFT, etc)

ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs

Apache License 2.0

659 stars 113 forks source link

Open alexhegit opened 5 months ago

alexhegit commented 5 months ago

Hello,

How to use llmperf to test ollama performance (TTFT, etc)?

Thanks, Alex

slyt commented 4 months ago

Ollama implements an OpenAI compatible API so you might be able to benchmark against that using the OpenAI benchmark instructions from the llmperf README. For the OPENAI_API_KEY you can pass in anything, but I think it's required.