search

ringsaturn / tz-benchmark

Continuous benchmark for timezone finders
https://ringsaturn.github.io/tz-benchmark/
MIT License
2 stars 1 forks source link

Unified benchmark #144

Open cyrusmsk opened 7 months ago

cyrusmsk commented 7 months ago

Hi, thanks for interesting project. Will you consider to make unified benchmark for different languages?

So it will be possible to see different solutions performance in one graph?

ringsaturn commented 7 months ago

It's an interesting point.

In profling cases, people prefer to use Google's pprof proto as a "standard" format.

And we have OTLP's open-telemetry/opentelemetry-proto as a unified data format for trace, log and metrics.

However I couldn't find a similar projects for this use cases. People in Google even use different tools for proto benchmarks. Google also has a repo: https://github.com/google/fleetbench seems close to what you mean, but still in early stage and not a universal way, you can take a look of it's giant proto files Message0.proto.

Maybe we can design a proto for this repo and see whether it counld be a unified way for benchmarks.

ringsaturn commented 7 months ago

After days of considering, it seems we can use current benchmark data and make a unified plot from it.

https://github.com/ringsaturn/tz-benchmark/blob/gh-pages/docs/data.js

However I knew little about JS&HTML.

cyrusmsk commented 6 months ago

Sorry for delay with the answer. I thought if it is possible to make a script which is using the same data as an input and different versions (rust, Python, go) will be able to read it and run to measure time. Then data could be presented on GitHub page yes.

ringsaturn commented 6 months ago

I believe that could be possible, but require some works to do.

ringsaturn commented 6 months ago

I'm testing bencher for this purpose. It seems they have unified many benchmark frameworks:

The first sample is tzfpy: https://github.com/ringsaturn/tzfpy/blob/main/.github/workflows/Bench.yml

You can view result from here: https://bencher.dev/perf/tzfpy

cyrusmsk commented 5 months ago

Thanks. Interesting service - bencher.dev It doesn't have adapter for the language I'm interested, but it has Hyperfine support. Maybe it will be possible to use it instead)