search

fullstorydev / solr-bench

Solr benchmarking and load testing harness
Apache License 2.0
17 stars 10 forks source link

Create periodic graphs via Jenkins (WIP). #30

Closed chatman closed 1 year ago

chatman commented 1 year ago

This contains general reorganization of the suites, ability to run them from any working directory (so that it can be run off Jenkins builds), and graph plotting.

chatman commented 1 year ago

FYI @gerlowskija, @patsonluk @noblepaul. I've merged the ishan/repeatable-jenkins branch to master now (through this PR).

patsonluk commented 1 year ago

Thank you for the great work @chatman ! I am going to merge that into my branch and test it out! 👍🏼

During the code merge in StressMain, which my branch allows restarting all nodes. There are several things that I wish to get some clarifications 😊 :

  1. For non local node, it appears the stop time will be 0 (ok as there's no way to split start and stop for non local one) but the start time will only account for the recovery time https://github.com/fullstorydev/solr-bench/blob/68682dac2cd42078c3c3c77a0bae6a28cd3142a5/src/main/java/StressMain.java#L308 ) ? I am thinking for non local node, perhaps the start time should be the restart time + recovery time?
  2. Since the total-time has accounted for the 5 secs ping on heap, i assume it simply point at how long does it take to run the benchmarking but has no/little significance on measuring the actual restart performance as the proper indicator of "real performance" should be node-shutdown and node-startup?
  3. What purposes do start-time and end-time serve (https://github.com/fullstorydev/solr-bench/blob/68682dac2cd42078c3c3c77a0bae6a28cd3142a5/src/main/java/StressMain.java#L326-L331). As inspected by the code logic, start-time appears to be the "wait time" between the "workflow is executed" to the "task (callable) starts execution" (kinda like waiting time?), while the end-time is "workflow is executed" to the "task (callable) finishes execution". They seem a bit confusing to me. If start-time is simply the time waited before the task is executed, can we use a different name? Not totally sure what end-time is for 😓 Perhaps they are useful for graphing purpose?
chatman commented 1 year ago

I am thinking for non local node, perhaps the start time should be the restart time + recovery time?

+1

Since the total-time has accounted for the 5 secs ping on heap, i assume it simply point at how long does it take to run the benchmarking but has no/little significance on measuring the actual restart performance as the proper indicator of "real performance" should be node-shutdown and node-startup?

Yes, the real metrics that are important are the node-startup and node-shutdown. I think the heap timings should also be one of the reported metrics, WDYT?

Perhaps they are useful for graphing purpose?

Exactly. It is used to demarcate the Solr metrics (heap, CPU etc.) on the metrics graphs by task. start-time is just the time offset since the beginning of the suite.

chatman commented 1 year ago

I am going to merge that into my branch and test it out! 👍🏼

Awesome.. Should we close/discard the https://github.com/fullstorydev/solr-bench/pull/33 in favour of the branch you'll merge? I'm +1 to that, if you think so.