fullstorydev / solr-bench

Solr benchmarking and load testing harness
Apache License 2.0
17 stars 10 forks source link

Some refactoring for graphing logic #35

Closed patsonluk closed 1 year ago

patsonluk commented 1 year ago

Description

This is a draft to explore some refactoring options on the existing result generation/graphing logic in solr-bench (stress.sh and the createGraph.py). @chatman @noblepaul Perhaps we can take some of these changes and apply to master if we find any of them useful.

Changes

StressMain.java

Minor refactoring of the result generation/export logic. Introduced interface ResultExporter (only implementation is FileResultExporter for now) that exports WorkflowResult.

stress.sh

  1. Generates a new meta-<test_name>.prop file with various meta info of a test. For example the git hash, branches, commit date, title, committer etc.
  2. The result files (results, metrics, config) will be placed in ${BASEDIR}/suites/results/${TEST_NAME}

New graph-scripts/create_graph_no_git.py

  1. Placed in a new folder graph-scripts, this makes it a bit easier to treat that as a separate python enabled module in the IDE
  2. The script has no dependency on git. It simply reads git (or any meta) info from a meta-xyz.prop file. This allows the script to focus on rendering logic. It is true that even on "external" mode, we will likely need to clone/read the git to generate such meta file, however, it's a bit nicer if such git operation only lives in one place (stress.sh)
  3. The script now separates the logic of "benchmark result parsing" from "html file generation". This allows extra "transformation" being done in between.
  4. Added a "--branches/-b" (required) parameter which:
    1. Allow specifying either a single branch name or multiple ones. The test logic scans all the meta files in the specified result directory and if the branches has any match with the -b param, then process the corresponding test result; Otherwise skip it.
    2. If -b has multiple values, then add an extra graph at the end by merging results from different branches into a single graph. Such logic can be made more generic in the future to compare/merge graphs of different "setups" (for example different # of nodes, different JDK/JVM options etc) of the same branch. image

Would be nice to get some feedbacks on this! 🙏🏼

chatman commented 1 year ago

Thanks, @patsonluk . I'll try this out tomorrow!

chatman commented 1 year ago

LGTM, these changes are very welcome! I followed through the logic, but haven't tested myself. But, this is the right direction. Please feel free to merge.

chatman commented 1 year ago

Thanks @patsonluk !