Framework to certify Pinot release (functional + performance)

[mayankshriv]

With the Pinot community growing, as well as Pinot evolving for the better, we anticipate a ton of features that would go into performance critical parts of the code (e.g. query execution, RT segment generation, etc). Given that Pinot's forte is its blazingly fast speed, we would like to ensure that the rapid feature development does not compromise on its performance and functionality. We need a framework using which any PR can be evaluated for performance/functionality. Some ideas:

Query Performance:

1. Create a few synthetic data sets with different characteristics that represent existing Pinot deployments across the community. We can probably enhance the existing data generator tool to server these requirements.
2. Generate a query set for a given data set to ensure desired level of selectivity.
3. Framework that runs a desired workload (qps) on these data sets with and without the changes from a PR, and reports the performance changes.

Realtime:

1. Consumption rate
2. Segment generation time

Ability to check on a variety of metrics, for example:

1. Latency p95/p99
2. Throughput
3. GC

https://docs.google.com/document/d/1QweQObrwGrcMR47dKzg1ZQpQkj4udHttkNm5kcMVBQ8/edit?usp=sharing

[siddharthteotia]

One way to use data sets which are nearly similar to production datasets is to use the Data Anonymizer tool (part of pinot-tools). This tool will generate anonymized data with similar characteristics (cardinality, distribution etc) as the given input data. It will also generate the corresponding queries.

So folks with Pinot deployments can do the following:

• Take your production dataset and corresponding queries.
• Provide them as input to the tool
• Tool will generate anonymized data and corresponding queries

Publish (hopefully allowed) the anonymized dataset and queries to the open source community so that it can be made part of the benchmark.

[kishoreg]

I like Sidd's idea, it will be good to have a framework where anyone can add their dataset and query set.

benchmark-root
-- /bin
    -- setup-cluster.sh
    -- run-benchmark.sh
-- cluster-spec.yaml. //can be a simple helm chart that can be used to spin up a pinot cluster in k8s
--   tables
      -- table-1
         -- table config
         -- schema
         -- Raw data
         -- ingestion spec
         -- query-runner-spec
         -- queries
         -- expected result
--  results

[FelixGV]

Hi Pinot community members :D !

I am not a Pinot user (yet!) but we have been thinking of a similar problem/solution (for other projects). One aspect that we're not sure about is how best to report historical performance data.

At a high level, we would like the ability to have a CI pipeline run some performance test on each commit, but we would also like the ability to take the output of those tests (latency metrics, etc.) and publish them to a dashboard where we could track the performance improvements and regressions of a given test across historical commits. We are wondering if there is a good off-the-shelf solution for this kind of CI-driven reporting of custom metrics, or whether we'd need to build something from scratch. I am also wondering if the historical reporting aspect is in scope of this issue, or if you are narrowing it down to the performance diff between the latest master commit and one PR.

Cheers!

[Jackie-Jiang]

@FelixGV Love the idea of keeping track of historical performance data

[FelixGV]

I got some suggestions from a couple of other folks on existing projects that do this:

• https://github.com/lightbend/benchdb
• https://github.com/jzillmann/jmh-visualizer/blob/master/README.md
• https://github.com/akarnokd/jmh-compare-gui
• https://github.com/blackboard/jmh-jenkins

I have not looked deeply at any of them, so IDK yet which is best, but hopefully some easy leverage is possible. Please LMK if you experiment with any of those! I'll also report back on our experience if we do.

[kishoreg]

We can always save the results in git (at least for releases). Saving historical numbers is a good feature to have but what's more important is to have the ability to run benchmark against a commit version. Once we have that ability, we can always re-generate the benchmark and save it.

The historical results can get outdated as we add/change benchmarks or change hardware.

Any off the shelf solution will just provide a way to invoke run a command. But the hard part is coming up with the actual queries/right metrics to measure - which is really project-specific.

IMO, it's better to write something custom that serves the project's purpose for now unless. WDYT

[FelixGV]

I agree that the core value of this work -- coming up with the actual queries/right metrics to measure -- is project-specific.

However, I think the generic part goes beyond just running commands. The reporting & visualization aspect could be non-trivial to build well, and moreover, the types of infrastructure projects that are very performance-sensitive are often specialized in things that are not very UI-focused, and therefore this visualization of historical data could be a weak point of an infra team.

I also agree that it would be nice for the tool to provide a way to backfill data for new benchmarks.

Anyway, like I said, if you do try some of these reporting tools, please report back. I'll do the same.

[shounakmk219]

So will this framework focus on doing performance test on the individual projects that are using pinot and track the performance difference when a new feature or some system tuning is performed, or it will focus on the pinot itself to track the performance difference for each pr?

[kishoreg]

@Jackie-Jiang can you please share your thoughts on this?

[Jackie-Jiang]

Here are some high level goals we want to achieve, feel free to leave comments: https://docs.google.com/document/d/1QweQObrwGrcMR47dKzg1ZQpQkj4udHttkNm5kcMVBQ8/edit?usp=sharing

[kishoreg]

Any volunteers to take this up. cc @shounakmk219 @KKcorps?

[mcvsubbu]

Just saw this issue, and wanted to flag a related issue: Functional testing Issue #4001

Sailesh Nankani https://www.linkedin.com/in/sailesh-nankani/ is working on that now

[mcvsubbu]

Here is the design doc for compatibility testing https://docs.google.com/document/d/1yNlvnLKDNUuyRWOKYYF01FWW9weYMGoaLRtU-CueciM/edit?ts=5e38bdd0#heading=h.ngrw7r8gso5w