search

ray-project / llmperf-leaderboard

Apache License 2.0
403 stars 10 forks source link

server spec for ray llm serving #12

Open Fizzbb opened 4 months ago

Fizzbb commented 4 months ago

Hi there,

I am wondering what hardware does ray use for serving in this llmperf leaderboard. Is it cpu or gpu? if it is GPU what's the model?

Thanks, Fizzbb

xieus commented 4 months ago

The dashboard uses llmperf benchmarking tool to run tests over different LLM APIs.

As mentioned in the llmperf benchmarking tool page, the backend of different API providers might vary widely, this is not a reflection on how the software runs on a particular hardware (could be GPUs, CPUs or custom ASICs). Hope this helps.