Not launching experiments on release launch day and the 3 follow days. It's not an issue for about 50% of experiments - but while the release is throttling in availability is not good for user enrollment and analysis. It can create low enrollment or even biasing issues - which are much worse on any release where there are delays in scaling up (at least a couple of times a year). It's best to just avoid all together.
NOTE: the exception may be if Nimbus decides to implement hold back studies - which by definition need to go out with the new release.
We want to cover this in on-boarding training... as there are types of things that will launch (like holdbacks) but we SHOULDN"T launch experiments.
shell1
commented
2 years ago
Not launching experiments on release launch day and the 3 follow days. It's not an issue for about 50% of experiments - but while the release is throttling in availability is not good for user enrollment and analysis. It can create low enrollment or even biasing issues - which are much worse on any release where there are delays in scaling up (at least a couple of times a year). It's best to just avoid all together.
NOTE: the exception may be if Nimbus decides to implement hold back studies - which by definition need to go out with the new release.
We want to cover this in on-boarding training... as there are types of things that will launch (like holdbacks) but we SHOULDN"T launch experiments.
┆Issue is synchronized with this Jira Task