While randomizing the execution order is probably the right thing to do for "most" cases, there are at least some cases whereby it's not desirable.
We wanted to use this library for a "dry-run" - we needed to perform the "experiment" flow before performing the "control" flow (executing the control flow first would invalidate the results of the experiement flow, but not vice-versa).
Describe the solution you'd like
It would be useful to have an option that disables randomiziation (control/experiment) and instead executes the trials in specified order
Experiement first, then control
Control first, then experiment
Describe alternatives you've considered
No alternatives exist that we can think of (outside of not using the library)
While randomizing the execution order is probably the right thing to do for "most" cases, there are at least some cases whereby it's not desirable.
We wanted to use this library for a "dry-run" - we needed to perform the "experiment" flow before performing the "control" flow (executing the control flow first would invalidate the results of the experiement flow, but not vice-versa).
Describe the solution you'd like It would be useful to have an option that disables randomiziation (control/experiment) and instead executes the trials in specified order
Describe alternatives you've considered No alternatives exist that we can think of (outside of not using the library)