mozilla / experimenter

A web application for managing user experiments for Mozilla Firefox.
https://experimenter.services.mozilla.com
Mozilla Public License 2.0
120 stars 186 forks source link

Removed FF min version values breaking experiments #4546

Closed data-sync-user closed 3 years ago

data-sync-user commented 3 years ago

[~pdehaan@mozilla.com] discovered that his experiment wasn't loading, and it looked like it was because his firefoxMinVersion was set to 80, which was recently removed.

> Expected a value of type \"NimbusExperimentFirefoxMinVersion\" but received: 80.!

We soon realized that many experiments in stage and production cannot load because they use versions that were deprecated.

Since bumping up the minimum version of or "no version"ing the broken experiments would invalidate them we should probably reinstate those older versions. Additionally, we should probably update the query to return the value from the DB and not use an enum, so when a version is removed it doesn't tank the experiment.

┆Issue is synchronized with this Jira Bug ┆Issue Number: EXP-915

pdehaan commented 3 years ago

Ref: https://stage.experimenter.nonprod.dataops.mozgcp.net/nimbus/test-cfr-experiment

Mozilla_Experimenter

GraphiQL for experimentBySlug(slug:"test-cfr-experiment") {...} returns:

{
  "errors": [
    {
      "message": "Expected a value of type \"NimbusExperimentFirefoxMinVersion\" but received: 80.!",
      "path": [
        "experimentBySlug",
        "firefoxMinVersion"
      ]
    }
  ],
  "data": {
    "experimentBySlug": {
      "id": 29,
      "owner": {
        "email": "aoprea@mozilla.com"
      },
      "status": "LIVE",
      "name": "test-cfr-experiment",
      "slug": "test-cfr-experiment",
      "publicDescription": "Testing cfr rapid experiments",
      "firefoxMinVersion": null
    }
  }
}
pdehaan commented 3 years ago

Internal Slack thread

Sounds like older versions should never be removed since it can break things unexpectedly. But I'm not sure what that looks like, since we don't want to keep "Firefox 80" permanently pinned to the top of a <select> field when we're currently on Firefox 103.0.2. Not sure if there is a way to hide older versions (prior to the current Release/ESR channel version) but still have them available for older experiments that targeted them.

jodyheavener commented 3 years ago

My thinking is we could have an enum of "current" versions provided through the config to populate the <select>, but use the value directly from the DB for the query. So the enum can be constantly changing while the stored value can do its own thing.

jaredlockhart commented 3 years ago

Oh yeah @jodyheavener that's not a bad idea, having two enums.

data-sync-user commented 3 years ago

➤ Lauren Zugai commented:

Can't find the PR from a quick search but I remember this being addressed and the link in the original issue comment now works.