Open alamb opened 1 year ago
I am currently focused on getting to the point where we can run the benchmarks repeatedly -- once I have it so I can easily run the benchmarks I will start working on running and collecting data over time.
I view the line protocol conversion as part of the story of over time conversion
I added details of a proposed design in the Proposed Design
section of the description of this ticket
@alamb would you mind to clarify a bit?
Are you planning to keep a collection kinda
<DataFusionVersion, Vec<Benchmark>>
?
If so the its also possible to backfill the history with archived benchmarks/versions
Apache Arrow uses the Conbench for similar purpose https://github.com/conbench/conbench
@comphead
Are you planning to keep a collection kinda <DataFusionVersion, Vec
>?
Yes -- that is basically what I have in mind. In my mind we would store it as lineprotocol and check it into a repo somewhere and then visualize it with existing tools (e.g. grafana and influxdb, which is what I know, but I am happy to use some other open source stack)
If so the its also possible to backfill the history with archived benchmarks/versions
Agree
Apache Arrow uses the Conbench for similar purpose https://github.com/conbench/conbench
Yes, I looked briefly into conbench (in fact there is some vestigal code in datafusion -- see https://github.com/apache/arrow-datafusion/tree/main/conbench and https://github.com/apache/arrow-datafusion/issues/5504 for details)
TLDR is I could not get it to work, and it seems as if the dev team went dormant(ish) so I didn't pursue it farther. If someone else can get it to work that would be great
Currently working on this. I have extended the existing compare.py
since it already reads in the existing json format well and I am producing rows of data in line protocol like this
benchmark,benchmark_version=32.0.0,datafusion_version=32.0.0,num_cpus=4 query="Query 1",row_count="4",elapsed="840.606454" 1697423932000
At this moment I am trying to setup an ingestion into influx using docker, but cannot quite seem to make the data available to visualize.
@Smurphy000 could you potentially push up what you have as a draft PR ? Maybe I can help with the "how to get this ingested / visualized" as I have more experience with that
Is your feature request related to a problem or challenge?
We want to have information about DataFusion's performance over time -- https://github.com/apache/arrow-datafusion/issues/5504. This is becoming more important as we work on more performance items / optimizations such as https://github.com/apache/arrow-datafusion/pull/5904
Currently the datafusion benchmarks in https://github.com/apache/arrow-datafusion/tree/main/benchmarks#datafusion-benchmarks can output the runs results as a JSON file.
I would like to use existing visualization systems (like timeseries databases).
Describe the solution you'd like
I would like to output the benchmark data optionally as LineProtocol https://docs.influxdata.com/influxdb/cloud-iox/reference/syntax/line-protocol/ so that it can be visualized by grafana / other systems that can handle line protocol
See https://grafana.com/docs/grafana-cloud/data-configuration/metrics/metrics-influxdb/push-from-telegraf/
Proposed Design
Write a python script, modeled after
compare.py
, that takes a performance json file and produces as output lineprotocolDesired output measurement:
benchmark
tags: details from run fields:query
iteration
,row_count
,elapsed_ms
timestamp: ns since epoch (I think that means multiply by 1000, but maybe by 1,000,000)Example output A line like this for each element in the
queries
array:Example input:
Here is a zip file with a bunch of example benchmark json files: results.zip
Describe alternatives you've considered
No response
Additional context
Related to https://github.com/apache/arrow-datafusion/issues/5504 tracking data over time