search

NVIDIA / nvbench

CUDA Kernel Benchmarking Library
Apache License 2.0
474 stars 63 forks source link

Tracking `nvbench` benchmark results #132

Open epompeii opened 1 year ago

epompeii commented 1 year ago

Is there any interest in tracking the results from nvbench?

I'm considering adding an adapter for nvbench to my continuous benchmarking tool, Bencher: https://github.com/bencherdev/bencher And I figured I should check in on whether there is already a preferred way to do this.

jrhemstad commented 1 year ago

Hey @epompeii, thanks for reaching out!

I hadn't heard of bencher before, but it looks great! A continuous benchmarking tool is something we've often talked about wanting to build, but have never got around to it.

I'd say we're definitely interested! What all is required?

epompeii commented 1 year ago

Thank you for the kind words @jrhemstad !

As for what's required, it depends if you want to self-host or use the hosted version.

To self-host, it's just two docker containers. One is the UI and one for the server + embedded DB. The DB can be set up to backup to S3 (this is the litestream docker image).

For the hosted version, you just have to sign up at https://bencher.dev and you should be good to go.

In either case, there is also the choice of compute. You can just use shared/GitHub Action runners. Depending on how you configure your thresholds, you can detect > ~50% performance regressions. If you want to be able to detect smaller regressions, then you would want to use a dedicated and preferably bare metal runner (like AWS Bare Metal). Eventually, I want to add bare metal runners as a feature to Bencher, so if you are interested in going this route, I would be more than willing to help with the effort.

jrhemstad commented 1 year ago

I'm guessing we'd want to self-host.

We will have self-hosted Action runners coming online in the next few months that we'd use for our performance tracking.

epompeii commented 1 year ago

Awesome!

I'm happy to help where I can to get things set up. My email is everett@bencher.dev or you are welcome to hop on our discord: https://discord.gg/yGEsdUh7R4 Please, feel free to ask me any questions or give candid feedback!

epompeii commented 5 months ago

Hey @jrhemstad, I just wanted to follow up, how are the self-hosted Action runners coming along?