Test initializer should default all subjects to out of all experiments.

howaboutwe / experimental

MIT License

20 stars 2 forks source link

Test initializer should default all subjects to out of all experiments. #29

Closed oggy closed 10 years ago

oggy commented 10 years ago

As described in the README, but was not working.

kchasel commented 10 years ago

@oggy on couples we're relying specifically on users not being defaulted out of all buckets to, over time, test all permutations of our tests put together. Is there an option provided to disable this feature in experiment settings?

oggy commented 10 years ago

Experimental.overrides.reset

I wouldn't recommend doing this for your primary build, though - nondeterministic test suites are a pain. I'd suggest at least restricting this to CI, if not a secondary test suite. And you'd want to report the actual buckets used somewhere so you can repro the failure in dev.