search

pytorch / benchmark

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.
BSD 3-Clause "New" or "Revised" License
880 stars 288 forks source link

How to get benchmark statistics? #2543

Open shink opened 1 week ago

shink commented 1 week ago

I'm building a CI to test some models on certain types of devices. I want get benchmark statistics like which model cases failed? which tests were skipped and why? These statistics will be used to generate a table like this:

Devices BERT_pytorch hf_GPT2
train eval train eval
CPU ✅ ✅ ✅ ✅
CUDA ✅ ✅ ✅ ✅
Foo ❌ (failed) ✅ ⚠️ (skipped) ✅

So how can I get benchmark statistics? Is there a recommended way to do this? Can anyone give suggestions? Thanks so much!

xuzhao9 commented 1 week ago

I suggest to write your own userbenchmark for this use case: https://github.com/pytorch/benchmark/blob/main/userbenchmark/ADDING_USERBENCHMARKS.md

shink commented 1 week ago

@xuzhao9 Thanks for your guidance! Actually, I'm building a CI to test some out-of-tree devices, such as Intel HPU, Ascend NPU, so I want to run some official model cases on them to prove the quality of these backends. I want to ask, is there any userbenchmarks that can be used directly? Or if I write my own benchmark, will it be merged?

shink commented 1 week ago

ah I think test_bench might be a good choice.

shink commented 1 week ago

But it only supports cpu and cuda. Will send a pull request to allow it to run on other devices.