Graphical output for benchmarks

[lykimq]

Summary

Benchmarks results that are easy to read and communicate to the team and outside of Marigold.

Motivation

As benchmark results depend on an important number of factors (see Issue #574), we need a easy way of visualizing them and provide reports when a new version of Deku is benchmarked as well.

Production

• Clean data
• Graphs
• Python notebooks?

[d4hines]

Prometheus provides basic timeseries visualization. See this example gathered from the data generated in https://github.com/marigold-dev/deku/pull/539:

Prometheus provides a simple expression language for doing calculations over the time series data. If you need more than this, it's possible to just export to e.g. Grafana, which provides more.

If you need even more than that (e.g. Python stats library or something), then, yes, we should export the prometheus data (either from Prometheus server, or scrape it separately) and then do what we need to do, but I think we can get quite far without this.

[ulrikstrid]

We can maybe make the benchmarks output something we can ingest in Prometheus or grafana

[d4hines]

We can maybe make the benchmarks output something we can ingest in Prometheus or grafana

I'm suggesting the reverse: gather the benchmark data by scraping our Prometheus endpoint.

[ulrikstrid]

I'm suggesting the reverse: gather the benchmark data by scraping our Prometheus endpoint.

@d4hines we're talking about different kinds of benchmarks. @lykimq wants to do benchmarks with the benchmarking lib(s) that exists in OCaml.

[d4hines]

@ulrikstrid,I guess I question the usefulness of microbenchmarks (i.e core_bench) used a priori. The true bottlenecks in the system are probably not what we expect, so I doubt our ability to write useful microbenchmarks in OCaml ahead of time. I think we need to set up a good e2e benchmarking system with profiling to help us identify the true bottlenecks. Once we identify the true bottlenecks, we can prevent regressions with microbenchmarks.

[lykimq]

Yes, I am thinking we will focus on the others:

• end-to-end
• profiling (perf, framegraph is the graph output; there are several tools support perf.data: speedscope, hotspot)
• micro-benchmarking, and other low level benchmarks (like finding the memory leaks, etc.)