Feature request: multi-line plots following the a/b/c structure of bench names

[rrnewton]

The gipeda UI is already fantastic -- I love the compare range widget.

Currently there's no way to get more than one line in a plot it looks like. One UI idea that might work here is to leverage the fact that benchmarks are already named hierarchically, e.g. containers/map/inserts.

I can currently click on the benchmark name and get the history just for the inserts benchmark. It would be great to click on map and get a group of all the map benchmarks together in one plot, or on containers for all the benchmarks.

The number of links and pages to generate here is still fixed and I hope not infeasible in the static page generation model. And I think this mechanism would have a good power/weight ratio. It can be used in several ways. For grouping, as above, but also for multiple-metrics per benchmark. If a data-producer wanted to enable this, it just names the metrics as follows:

containers/map/inserts/time
containers/map/inserts/alloc
...

That way one could click on the inserts link to get all the metrics for a specific benchmark.

Maybe a heuristic is appropriate to limit the page generation: if there is too much data or too many lines, don't generate the page and leave the link dead.

[nomeata]

Worthwhile ideas. As a first step, I’d like to implement displaying two or more benchmarks on the same line, with a selection widget on the graph page. Then we can see if the ability to quickly select a whole set of benchmarks is required.

This does not require changes to the static generation, as this combination would happen purely on the client side.

Note that the hierarchy with / is currently not used anywhere in gipeda, so this would add semantics to something that was insignificant before, so I’m not sure about that detail. Benchmark groups are more likely the right abstraction for such grouping.

[rrnewton]

Yeah, you would want the / to be configurable. Yes, it should be optional. And yes, it would be adding semantics to names... but its awful tempting because the information is already mostly there waiting to be mined.

Do you mean benchmark groups at the criterion level? Or defining some new level of benchmark groups in the gipeda settings? That sounds like it would be hard to do for a large number of preexisting repositories.

I think the alternative, to avoid giving semantics to names, is to include one more field of information, i.e. the metric name: (BenchmarkName, MetricName, Value). @sgraf812 wants to limit scope creep, understandably. But just including that one extra field -- even if it is hardcoded to "time" for now -- could be helpful downstream.

For example, I think we or someone else, if we are using the cloben script, will need to soon extend it to not throw away the extra information that criterion can report: numGCs, bytesAllocated, bytesCopied, gcWallSeconds. And whether through naming scheme or an extra field, it would be great to have some place to put this data. Likewise, if we dump data hsbencher->gipeda, we need to pick how to encode these multiple "columns" of data. We can squish it into the name, or do something else.

If there is a willingness to allow an extra field, I don't think it would be heavy lifting. It would just mean a 3-column CSV file rather than a 2-column one. To me it seem like you guys are already doing the hard parts -- UI, tool integration, Javascript business, robust log parsing. A slight addition to the data model seems comparatively easy to me ;-).

[nomeata]

Or defining some new level of benchmark groups in the gipeda settings?

Not new, rather the existing ones that you see on https://perf.haskell.org/ghc/#graphs

And whether through naming scheme or an extra field, it would be great to have some place to put this data.

There is no problem storing the data (i.e. the results) in the CSV file right now. It seems to me, though, that you want to also store meta-data about the benchmarks, e.g. its “type” so to say, in the CSV. I’m not sure I like that, as it makes a few things more complicated – what if different revisions assign different types to the same benchmark?

I’d rather have the CSV contain the raw data and have extra information (type, whether smaller is better, group, thresholds, importance) about the benchmark in a separate file. See https://github.com/nomeata/gipeda/blob/master/ghc/settings.yaml for how that looks today.

This also allows adjustments of these settings independent of actually generating new data, and it applies also to previous data sets.

[rrnewton]

Got it. I spoke too soon as I hadn't understood the current settings.yaml format... This looks pretty flexible.

So it IS using information embedded in names and regular expressions against names.

Your proposal on top of that mechanism is to be able to "union" arbitrary subsets of these graphs to see all multiple lines at once? That actually sounds pretty great, if that degree of client-side computation is acceptable.

[rrnewton]

This settings.yaml format is also great because it means that future growth in, e.g., cloben can be well accommodated without any changes to gipeda.

[nomeata]

So it IS using information embedded in names and regular expressions against names.

But here “it” is per-project user specified “code”, and not something that is hard-coded in gipeda.

Your proposal on top of that mechanism is to be able to "union" arbitrary subsets of these graphs to see all multiple lines at once? That actually sounds pretty great, if that degree of client-side computation is acceptable.

I believe it is; the graphs are already client-side generated. It “just” needs the UI work for that.