fullstorydev / solr-bench

Solr benchmarking and load testing harness
Apache License 2.0
17 stars 10 forks source link

Unhandled exceptions should interrupt all actions in `ControlledExecutor` #89

Closed patsonluk closed 9 months ago

patsonluk commented 9 months ago

Description

Current issues with exception handling

  1. When UploadDoc failed to upload documents, it would print exception but keep going. This might not make sense if the exception is non-recoverable, for example collection not found
  2. ControlledExecutor would keep executing all actions and would only throw ExecutionException after finishing all executions. We would rather have it interrupt and stop all actions for unhandled exception instead.

Solution

  1. Introduced new flag interrupt-on-failure (default true) under the index benchmark configuration. If true, we would now throw RuntimeException if the response code is not success in UploadDoc.
  2. Refactored the ControlledExecutor to achieve the desired behaviors a. Adjusted the terminology to avoid confusion. Now that task is the one that is defined in the configuration in the workflow, each task can have its own ControlledExecutor, the task would also provide an "action supplier", which supplies the "action" to be executed. ControllerExecutor regulates the concurrencies, rate and/or duration of the execution b. Added logic to monitor each submitted action (the Future), if any of them throws exception, we would cancel (interrupt) all the other actions being executed by this executor c. Introduced the StatusChecker which better tracks the "status" of execution, it also signals whether the executor should be stopped and block if necessary (rate and back pressure regulation)

Test

Tested locally to ensure things work as expected after the changes:

  1. RPM and duration still work as expected
  2. Verify the debug message from the Progress Timer
  3. Force exception from UploadDocs, verify that all jobs are interrupted in the Indexing task and the following query tasks would not be executed anymore