search

huggingface / optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
Apache License 2.0
236 stars 41 forks source link

Add support for intel in leaderboard #248

Closed baptistecolle closed 3 weeks ago

baptistecolle commented 1 month ago

This pr add support for intel on llm-perf-leaderboard. We target the intel xeon 4th generation sapphire rapids

This adds also benchmark on different backends

This is related to the PR that add the results to the leaderboard - https://huggingface.co/spaces/optimum/llm-perf-leaderboard/discussions/32

(ORT and OpenVino will come in another PR)

baptistecolle commented 3 weeks ago

There is also a PR that add the results to the leaderboard - https://huggingface.co/spaces/optimum/llm-perf-leaderboard/discussions/32

I have left it as a draft for now, as I am waiting for this PR to be merged to ensure that the naming conventions and other details match. However, please let me know if you have any feedback on it already

IlyasMoutawwakil commented 3 weeks ago

good overall, but I would like to keep a difference between hardware and software here, the benchmarking scripts are independent from the hardware, for example the same cuda_pytorch script can run on amd rocm gpus, the cpu_pytorch can run on an intel cpu or arm cpu or amd cpu, so I don't see a need to make these scripts related to any hardware provider. on the leaderboard/frontend side yes it makes sense to indicate the provider, but no need in the backend (the same script will run on an amd cpu). In the workflow it might make sense but then again there's no need when the "MACHINE" name already tells you what hardware it's running on.