mozilla / experimenter

A web application for managing user experiments for Mozilla Firefox.
https://experimenter.services.mozilla.com
Mozilla Public License 2.0
114 stars 176 forks source link

Update enrollment conflict error messages #10353

Open danielkberry opened 4 months ago

danielkberry commented 4 months ago

There are a few situations where conflicts will reduce the eligible population for an upcoming experiment. We currently have warnings, but the messages are vague. We should instead update the message text to:

The following … and may reduce the eligible population for your experiment which may result in reduced statistical power & precision. Please check that the configured population proportion has accounted for this.

Where depends on the type of conflict:

  1. An experiment excludes other live deliveries: “experiments are excluded by your experiment ”
  2. Live experiments exist in previous iteration of a namespace: “experiments are LIVE on a previous namespace iteration”
  3. Live multi-feature experiments exist for this feature: “multi-feature experiments are LIVE”

┆Issue is synchronized with this Jira Task

danielkberry commented 4 months ago

CC @jaredlockhart