When UploadDoc failed to upload documents, it would print exception but keep going. This might not make sense if the exception is non-recoverable, for example collection not found
ControlledExecutor would keep executing all actions and would only throw ExecutionException after finishing all executions. We would rather have it interrupt and stop all actions for unhandled exception instead.
Solution
Introduced new flag interrupt-on-failure (default true) under the index benchmark configuration. If true, we would now throw RuntimeException if the response code is not success in UploadDoc.
Refactored the ControlledExecutor to achieve the desired behaviors
a. Adjusted the terminology to avoid confusion. Now that task is the one that is defined in the configuration in the workflow, each task can have its own ControlledExecutor, the task would also provide an "action supplier", which supplies the "action" to be executed. ControllerExecutor regulates the concurrencies, rate and/or duration of the execution
b. Added logic to monitor each submitted action (the Future), if any of them throws exception, we would cancel (interrupt) all the other actions being executed by this executor
c. Introduced the StatusChecker which better tracks the "status" of execution, it also signals whether the executor should be stopped and block if necessary (rate and back pressure regulation)
Test
Tested locally to ensure things work as expected after the changes:
RPM and duration still work as expected
Verify the debug message from the Progress Timer
Force exception from UploadDocs, verify that all jobs are interrupted in the Indexing task and the following query tasks would not be executed anymore
Description
Current issues with exception handling
UploadDocfailed to upload documents, it would print exception but keep going. This might not make sense if the exception is non-recoverable, for example collection not foundControlledExecutorwould keep executing all actions and would only throwExecutionExceptionafter finishing all executions. We would rather have it interrupt and stop all actions for unhandled exception instead.Solution
interrupt-on-failure(defaulttrue) under the index benchmark configuration. Iftrue, we would now throwRuntimeExceptionif the response code is not success inUploadDoc.ControlledExecutorto achieve the desired behaviors a. Adjusted the terminology to avoid confusion. Now thattaskis the one that is defined in the configuration in the workflow, each task can have its ownControlledExecutor, the task would also provide an "action supplier", which supplies the "action" to be executed.ControllerExecutorregulates the concurrencies, rate and/or duration of the execution b. Added logic to monitor each submitted action (theFuture), if any of them throws exception, we would cancel (interrupt) all the other actions being executed by this executor c. Introduced theStatusCheckerwhich better tracks the "status" of execution, it also signals whether the executor should be stopped and block if necessary (rate and back pressure regulation)Test
Tested locally to ensure things work as expected after the changes: