search

ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
308 stars 22 forks source link

Question: export option for full fidelity Histogram? #82

Closed heckj closed 1 year ago

heckj commented 1 year ago

I was looking through the export options, spotted the JMH and .tsv exports, but one of the things I wanted to explore was taking the full-fidelity Histogram from package-histogram, loading the file back in using Histogram's Codable conformance, and exploring some local visualization with it (SwiftUI Charts, etc).

I'm not familiar with .tsv (other than an assumption is stands for "time series values") and wanted to ask - is that a more sane path for reloading into my own Histogram instance, or would it be reasonable to extend the exports to drop out a JSON from Histogram's codable? I'm reading and learning on package-histogram, but not 100% on all the pieces and parts, and what's legit for reproducing it by reading in a file.

I'd presumed, but not yet traced, that dumping a ton of values into even a HdrHistogram was fundamentally lossy - and that it didn't store the entirety of all the values submitted to it. I'm happy to do the PR to enable this thing I want, figured I'd best ask first - maybe there's a path that's easily there I'm overlooking.

hassila commented 1 year ago

https://github.com/ordo-one/package-histogram/issues/2

Would be the proper solution I think - the upstream hdrhistogram supports a binary format for this that should be supported in that context.

The histogram basically stores a count of all values with the specified precision so it's possible to get out all values with an iterator (as is done by eg. Jmh export).

The codable support for histogram works ("all" samples are encoded logically), but it's not very efficient as the vector of counts (not individual samples) can be reasonably large depending on the magnitude and precision of captured values and json is... json.

hassila commented 1 year ago

(Adding codable to histogram was really a short term fix for the use of benchmark)

That being said, I don't see any issue with adding an export option for the histograms codable representation- "--format=histogram-json" or "histogram-codable"? Happily take a PR for it.

Later we can add the binary representation "hgrm".

Side note is that the tsv export should be improved to support multiple baselines. It's primarily intended to be used as input to youplot or gnuplot.

Also need to standardize file naming a bit I think - let me think a bit, perhaps BenchmarkResult should get a function that returns a standardized file name for the multi export case.

hassila commented 1 year ago

Fixed by #84