Schema Change for Adding Rollouts

[jaredlockhart]

Welcome Change Captain! ⛵️

When making schema changes, follow this guidance to avoid breaking changes:

• Rather than modifying an existing field, consider adding a new field with a new name and later deprecating the old field when it is no longer used

  • When modifying a field by adding and deprecating, document:
  • The expected behaviour of the old field in terms of the new
  • The expected behaviour when both fields are present. (e.g. in the presence of both fields, the new old field should be ignored)

• When deprecating an existing field, consider making it optional/nullable rather than removing it altogether and then removing it entirely when all clients/active experiments no longer depend on it

Proposing Schema Changes

• [x] Assign this issue to yourself
• [x] Add a comment below describing the changes at a high level, including all new/deprecated fields, their types, and their uses
  • [x] Link to any related proposal documentation
• [x] Create a PR which introduces the schema changes and includes the following
  • [x] Changes to relevant types
  • [x] Changes to their associated tests
  • [x] New sample data demonstrating their use for each of desktop/mobile
• [x] Send an email to the Services RFCs Mailing List with a link to the PR requesting feedback
• [x] Receive feedback, update the PR as necessary, and commit it
• [x] Create a release including the new changes
• [x] File tickets/epics for all Nimbus clients to implement the changes
  • [x] Nimbus Desktop
  • [x] Nimbus Mobile
  • [x] Add all filed tickets/epics in a comment below
• [x] For deprecated fields, file a ticket in Nimbus Engineering to add a warning to the Nimbus UI that the field will be deprecated and provide guidance about how experiment/feature owners should migrate away from their dependence on the deprecated field

When Client Changes Are Complete

• [x] File tickets/epics for all Nimbus Clients for the new changes to be QAed by supplying the new sample data to the relevant (desktop/mobile) client(s) and verifying the intended behaviour
  • [x] Nimbus Desktop
  • [x] Nimbus Mobile
  • [x] Add all filed tickets/epics in a comment below

When Client QA is Complete

• [x] Add a comment below with a QA report that includes
  • [x] Which version of each client application was tested
  • [x] Link to any bugs which were detected in testing

When Bugs Are Resolved and Clients Are Released

• [ ] Comment below indicating which version of each client application includes the changes

  • Hint: this is much easier to do by noting the last version of the client application that didn't have the change at the time of making the change

• [x] File tickets/epics in Nimbus Engineering to implement the changes in Experimenter and add them in a comment below, including:

  • [x] Changes to relevant APIs
  • [x] Changes to UI
  • Note for deprecated fields
  • [x] First prevent the usage of deprecated fields in experiments targeting versions where those fields are no longer supported
  • [x] Then make sure no currently active experiments still depend on them before removing them from UI/APIs
  • Leave the fields in the database so that historical data is preserved and the change can be easily reverted if necessary

• [ ] File tickets/epics in Jetstream to adapt it to any incoming changes that affect its analysis

  • Note changing Jetstream and Experimenter may cause conflicts so try to coordinate both changes to go out as close together as possible

When Experimenter Changes Are Complete

• [ ] You're done! Close this ticket.

[lmorchard]

Add a comment below describing the changes at a high level, including all new/deprecated fields, their types, and their uses Link to any related proposal documentation

Related documentation is mainly this Nimbus Rollouts Technical Plan

In particular, adding a per-experiment isRollout boolean property to the experiment DTO per the "System Design, Mega MVP 2" section"

• Mega MVP 2
  • Experimenter
    • Add is_rollout bool to experiment
    • Set is rollout bool on DTO

[lmorchard]

Create a PR which introduces the schema changes

https://github.com/mozilla/nimbus-shared/pull/179

Will see if any tests are relevant here

[lmorchard]

File tickets/epics for all Nimbus clients to implement the changes Nimbus Mobile

Looks like the ticket for Mobile has already been filed and implemented

Nimbus Desktop

And filed & in-progress for desktop as well

[heres-maria-sv-inactive]

Considering the changes recently introduced by https://github.com/mozilla/nimbus-shared/issues/182, the testing recipes might need to take into account the multifeature changes as well. For testing we would also need to use both single feature and multifeature experiments that use the new schema for Rollouts.

[lmorchard]

Looks like the ticket for QA on Desktop and Mobile is here:

https://mozilla-hub.atlassian.net/browse/EXP-1868

[lmorchard]

Merged the schema change, but have not yet made a release. We have some sample data candidates here that I'd like to wait for before the release:

https://github.com/mozilla/nimbus-shared/pull/179#issuecomment-1012215766

But also, since we've gone back & forth so much on that data, I'd also like to wait until QA for the clients is complete so we know exactly what data was practically useful.

[heres-maria-sv-inactive]

Current testing data based on https://github.com/mozilla/nimbus-shared/pull/179#issuecomment-1012215766 (feature config values - e.g. loadMore -> true/false - and channel may change as testing is performed. Filters that can be found on Stage may be added to change the order in which we enroll in the available recipes):

Desktop:

Desktop Rollout with pocketNewtab enabled

``` { "slug": "firefox-desktop-rollout-test-final", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": true, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "descLines": 2, "newFooterSection": true, "lastCardMessageEnabled": false }, "enabled": true, "featureId": "pocketNewtab" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": "2021-10-26", "targeting": "true", "featureIds": [ "pocketNewtab" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-rollout", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Rollout Test Pocket", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for pocketNewTab", "id": "firefox-desktop-rollout-test-final", "last_modified": 1642427406371 } ```

Desktop Rollout with privatebrowsing disabled

``` { "slug": "firefox-desktop-rollout-test-final-2", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "privatebrowsing" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": "2021-10-26", "targeting": "true", "featureIds": [ "privatebrowsing" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-privatebrowsing-rollout", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Rollout Test 2", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for privatebrowsing", "id": "firefox-desktop-rollout-test-final-2", "last_modified": 1639560912253 } ```

Desktop Experiment with isRollout false pocketNewtab enabled / disabled

``` { "slug": "firefox-desktop-rollout-experiment1", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "pocketNewtab" } ] }, { "slug": "treatment", "ratio": 1, "feature": { "value": {}, "enabled": true, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "compactLayout": true, "newFooterSection": false, "lastCardMessageEnabled": false }, "enabled": true, "featureId": "pocketNewtab" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": "2021-10-26", "targeting": "!homePageSettings.isDefault && homePageSettings.isCustomUrl && homePageSettings.urls[.host == 'google.com']|length > 0", "featureIds": [ "pocketNewtab" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-5", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Experiment 1 Pocket enabled", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment 1 enabled for pocketNewTab", "id": "firefox-desktop-rollout-experiment1", "last_modified": 1642430566908 } ```

Desktop Multifeature Experiment with isRollout false pocketNewtab and privatebrowsing enabled / disabled

``` { "slug": "firefox-desktop-rollout-experimentm1", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "pocketNewtab" }, { "value": {}, "enabled": false, "featureId": "privatebrowsing" } ] }, { "slug": "treatment", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "descLines": 3, "compactLayout": true, "newFooterSection": true, "lastCardMessageEnabled": true }, "enabled": true, "featureId": "pocketNewtab" }, { "value": { "infoBody": "This is some text", "infoIcon": "chrome://branding/content/about-logo.png", "infoTitle": "Hello world", "promoTitle": "Promo title", "infoEnabled": true, "infoLinkUrl": "http://foo.mozilla.com/%LOCALE%", "infoLinkText": "This is a link", "promLinkText": "Promo link", "promoEnabled": true, "promoLinkUrl": "https://vpn.mozilla.org/?utm_source=firefox-browser&utm_medium=firefox-%CHANNEL%-browser&utm_campaign=private-browsing-vpn-link&utm_experiment=private-browsing-test&utm_branch=treatment" }, "enabled": true, "featureId": "privatebrowsing" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": "2021-10-26", "targeting": "!('browser.newtabpage.activity-stream.showSearch'|preferenceIsUserSet)", "featureIds": [ "pocketNewtab", "privatebrowsing" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-privatebrowsing-1", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Experiment 1 multifeature enabled", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment 1 multifeature enabled", "id": "firefox-desktop-rollout-experimentm1", "last_modified": 1642431110139 } ```

Firefox Android:

Android Rollout with homescreen enabled

``` { "slug": "firefox-android-rollout-test-final", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": false, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [ { "slug": "default-browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Rollout Test 1", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android ROllout Homescreen", "id": "firefox-android-rollout-test-final", "last_modified": 1639991916998 } ```

Android Rollout with homescreen disabled

``` { "slug": "firefox-android-rollout-test-final-2", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] } ], "outcomes": [ { "slug": "default-browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Rollout Test 2", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Rollout Homescreen disabled", "id": "firefox-android-rollout-test-final-2", "last_modified": 1639991916998 } ```

Android Experiment with isRollout false homescreen enabled /disabled

``` { "slug": "firefox-android-homepage-experiment-1", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] }, { "slug": "treatment", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": true, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-3", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Experiment 1", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Homepage on with a difference", "id": "firefox-android-homepage-experiment-1", "last_modified": 1639991916998 } ```

Android Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

``` { "slug": "firefox-android-homepage-experiment-m1", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": { "sections-enabled": {} }, "enabled": false, "featureId": "homescreen" }, { "value": {}, "enabled": false, "featureId": "fenix-default-browser" } ] }, { "slug": "default_browser_settings_menu", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": true, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" }, { "value": {}, "enabled": true, "featureId": "fenix-default-browser" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "fenix-default-browser", "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-fenix-default-browser-homescreen-release-7", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Experiment Multifeature 1", "referenceBranch": "default_browser_settings_menu", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Homepage on with a single difference multifeature", "id": "firefox-android-homepage-experiment-m1", "last_modified": 1639991916998 } ```

Firefox iOS:

iOS Rollout with homescreen enabled

``` { "slug": "firefox-ios-rollout-test-final", "appId": "org.mozilla.ios.Firefox", "appName": "firefox_ios", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": { "jumpBackIn": false, "libraryShortcuts": true, "recentlySaved": true }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.ios.Firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Rollout 1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for homescreen enabled feature", "id": "firefox-ios-rollout-test-final", "last_modified": 1639561341272 } ```

iOS Rollout with homescreen disabled

``` { "slug": "firefox-ios-rollout-test-final-2", "appId": "org.mozilla.ios.Firefox", "appName": "firefox_ios", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "onboarding-default-browser" } ] } ], "outcomes": [ { "slug": "default_browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "onboarding-default-browser" ], "application": "org.mozilla.ios.Firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-onboarding-default-browser-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Rollout 2", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for onboarding-default-browser disabled feature", "id": "firefox-ios-rollout-test-final-2", "last_modified": 1639561341272 } ```

iOS Experiment with isRollout false homescreen enabled / disabled

``` { "slug": "firefox-ios-homepage-experiment-1", "appId": "org.mozilla.ios.Firefox", "appName": "firefox_ios", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] }, { "slug": "treatment", "ratio": 1, "features": [ { "value": { "jumpBackIn": false, "libraryShortcuts": true, "recentlySaved": true }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.ios.Firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-release-3", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Single Feature Experiment 1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment for homescreen enabled feature", "id": "firefox-ios-homepage-experiment-1", "last_modified": 1639561341272 } ```

iOS Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

``` { "slug": "firefox-ios-homepage-experiment-m1", "appId": "org.mozilla.ios.Firefox", "appName": "firefox_ios", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" }, { "value": {}, "enabled": false, "featureId": "onboarding-default-browser" } ] }, { "slug": "default_browser_newtab_banner", "ratio": 1, "features": [ { "value": { "jumpBackIn": false, "libraryShortcuts": true, "recentlySaved": true }, "enabled": true, "featureId": "homescreen" }, { "value": {}, "enabled": true, "featureId": "onboarding-default-browser" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen", "onboarding-default-browser" ], "application": "org.mozilla.ios.Firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-onboarding-default-browser-release-3", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Single Feature Experiment Multifeature 1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment multifeature for homescreen + onboarding-default-browser enabled features", "id": "firefox-ios-homepage-experiment-m1", "last_modified": 1639561341272 } ```

[heres-maria-sv-inactive]

Final testing data used:

Desktop:

Desktop Rollout with pocketNewtab enabled

``` { "slug": "firefox-desktop-rollout-test-final", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": true, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "descLines": 2, "newFooterSection": true, "lastCardMessageEnabled": false }, "enabled": true, "featureId": "pocketNewtab" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": "2021-10-26", "targeting": "true", "featureIds": [ "pocketNewtab" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-rollout", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Rollout Test Pocket", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for pocketNewTab", "id": "firefox-desktop-rollout-test-final", "last_modified": 1642427406371 } ```

Desktop Rollout with privatebrowsing disabled

``` { "slug": "firefox-desktop-rollout-test-final-2", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "privatebrowsing" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": "2021-10-26", "targeting": "true", "featureIds": [ "privatebrowsing" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-privatebrowsing-release-rollout", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Rollout Test 2", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for privatebrowsing", "id": "firefox-desktop-rollout-test-final-2", "last_modified": 1642415370678 } ```

Desktop Experiment with isRollout false pocketNewtab enabled / disabled

``` { { "slug": "firefox-desktop-rollout-experiment1", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "pocketNewtab" } ] }, { "slug": "treatment", "ratio": 1, "feature": { "value": {}, "enabled": true, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "compactLayout": true, "newFooterSection": false, "lastCardMessageEnabled": false }, "enabled": true, "featureId": "pocketNewtab" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": "2021-10-26", "targeting": "true", "featureIds": [ "pocketNewtab" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-5", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Firefox Desktop Experiment 1 Pocket enabled", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment 1 enabled for pocketNewTab", "id": "firefox-desktop-rollout-experiment1", "last_modified": 1643181726679 } ```

Desktop Multifeature Experiment with isRollout false pocketNewtab and privatebrowsing enabled / disabled

``` { "slug": "firefox-desktop-rollout-experimentm1", "appId": "firefox-desktop", "appName": "firefox_desktop", "channel": "", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": {}, "enabled": false, "featureId": "pocketNewtab" }, { "value": {}, "enabled": false, "featureId": "privatebrowsing" } ] }, { "slug": "treatment", "ratio": 1, "feature": { "value": {}, "enabled": false, "featureId": "unused-feature-id-for-legacy-support" }, "features": [ { "value": { "loadMore": true, "descLines": 3, "compactLayout": true, "newFooterSection": true, "lastCardMessageEnabled": true }, "enabled": true, "featureId": "pocketNewtab" }, { "value": { "infoBody": "This is some text", "infoIcon": "chrome://branding/content/about-logo.png", "infoTitle": "Hello world", "promoTitle": "Promo title", "infoEnabled": true, "infoLinkUrl": "http://foo.mozilla.com/%LOCALE%", "infoLinkText": "This is a link", "promLinkText": "Promo link", "promoEnabled": true, "promoLinkUrl": "https://vpn.mozilla.org/?utm_source=firefox-browser&utm_medium=firefox-%CHANNEL%-browser&utm_campaign=private-browsing-vpn-link&utm_experiment=private-browsing-test&utm_branch=treatment" }, "enabled": true, "featureId": "privatebrowsing" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": "2021-10-26", "targeting": "!('browser.newtabpage.activity-stream.showSearch'|preferenceIsUserSet)", "featureIds": [ "pocketNewtab", "privatebrowsing" ], "application": "firefox-desktop", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "firefox-desktop-pocketNewtab-privatebrowsing-1", "randomizationUnit": "normandy_id" }, "schemaVersion": "1.6.2", "userFacingName": "Experiment 1 multifeature enabled", "referenceBranch": "control", "proposedDuration": 63, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment 1 multifeature enabled", "id": "firefox-desktop-rollout-experimentm1", "last_modified": 1642431110139 } ```

Firefox Android:

Android Rollout with homescreen enabled

``` { "slug": "firefox-android-rollout-testfinal", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": false, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [ { "slug": "default-browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Rollout Test 1", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android ROllout Homescreen", "id": "firefox-android-rollout-testfinal", "last_modified": 1642776439844 } ```

Android Rollout with homescreen disabled

``` { "slug": "firefox-android-rollout-test-final-2", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] } ], "outcomes": [ { "slug": "default-browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Rollout Test 2", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Rollout Homescreen disabled", "id": "firefox-android-rollout-test-final-2", "last_modified": 1642778041310 } ```

Android Experiment with isRollout false homescreen enabled /disabled

``` { "slug": "firefox-android-homepage-experiment-1", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] }, { "slug": "treatment", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": true, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-homescreen-release-3", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Experiment 1", "referenceBranch": "control", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Homepage on with a difference", "id": "firefox-android-homepage-experiment-1", "last_modified": 1642777734710 } ```

Android Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

``` { "slug": "firefox-android-homepage-experiment-m1", "appId": "org.mozilla.firefox", "appName": "fenix", "channel": "release", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" }, { "value": {}, "enabled": false, "featureId": "fenix-default-browser" } ] }, { "slug": "default_browser_settings_menu", "ratio": 1, "features": [ { "value": { "sections-enabled": { "pocket": true, "topSites": false, "jumpBackIn": true, "recentlySaved": false, "recentExplorations": false } }, "enabled": true, "featureId": "homescreen" }, { "value": {}, "enabled": true, "featureId": "fenix-default-browser" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "fenix-default-browser", "homescreen" ], "application": "org.mozilla.firefox", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "fenix-fenix-default-browser-homescreen-release-7", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "Homepage Android Experiment Multifeature 1", "referenceBranch": "default_browser_settings_menu", "proposedDuration": 45, "isEnrollmentPaused": false, "proposedEnrollment": 28, "userFacingDescription": "Android Homepage on with a single difference multifeature", "id": "firefox-android-homepage-experiment-m1", "last_modified": 1643117825617 } ```

Firefox iOS:

iOS Rollout with homescreen enabled

``` { "slug": "firefox-ios-rollout-test-final", "appId": "org.mozilla.ios.Fennec", "appName": "firefox_ios", "channel": "nightly", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": { "sections-enabled": { "jumpBackIn": false, "libraryShortcuts": true, "recentlySaved": true } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.ios.Fennec", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-nightly-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Rollout 1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for homescreen enabled feature", "id": "firefox-ios-rollout-test-final", "last_modified": 1639561341272 } ```

iOS Rollout with homescreen disabled

``` { "slug": "firefox-ios-rollout-test-final2", "appId": "org.mozilla.ios.Fennec", "appName": "firefox_ios", "channel": "nightly", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] } ], "outcomes": [ { "slug": "default_browser", "priority": "primary" } ], "arguments": {}, "isRollout": true, "probeSets": [], "startDate": null, "targeting": "true", "featureIds": [ "homescreen" ], "application": "org.mozilla.ios.Fennec", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-nightly-rollout", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "iOS Rollout 2", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Rollout for homescreen disabled feature", "id": "firefox-ios-rollout-test-final2", "last_modified": 1643212963537 } ```

iOS Experiment with isRollout false homescreen enabled / disabled

``` { "slug": "firefox-ios-homepage-experiment-1", "appId": "org.mozilla.ios.Fennec", "appName": "firefox_ios", "channel": "nightly", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" } ] }, { "slug": "treatment", "ratio": 1, "features": [ { "value": { "sections-enabled": { "jumpBackIn": false, "recentlySaved": true, "libraryShortcuts": true } }, "enabled": true, "featureId": "homescreen" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "is_already_enrolled || days_since_install < 7", "featureIds": [ "homescreen" ], "application": "org.mozilla.ios.Fennec", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-nightly-10", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "firefox-ios-homepage-experiment-1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Experiment for homescreen enabled feature", "id": "firefox-ios-homepage-experiment-1", "last_modified": 1643181387370 } ```

iOS Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

``` { "slug": "firefox-ios-homepage-experiment-m1", "appId": "org.mozilla.ios.Fennec", "appName": "firefox_ios", "channel": "nightly", "endDate": null, "branches": [ { "slug": "control", "ratio": 1, "features": [ { "value": {}, "enabled": false, "featureId": "homescreen" }, { "value": {}, "enabled": false, "featureId": "onboarding-default-browser" } ] }, { "slug": "default_browser_newtab_banner", "ratio": 1, "features": [ { "value": { "sections-enabled": { "jumpBackIn": false, "recentlySaved": true, "libraryShortcuts": false } }, "enabled": true, "featureId": "homescreen" }, { "value": {}, "enabled": true, "featureId": "onboarding-default-browser" } ] } ], "outcomes": [], "arguments": {}, "isRollout": false, "probeSets": [], "startDate": null, "targeting": "is_already_enrolled || days_since_install < 7", "featureIds": [ "homescreen", "onboarding-default-browser" ], "application": "org.mozilla.ios.Fennec", "bucketConfig": { "count": 10000, "start": 0, "total": 10000, "namespace": "ios-homescreen-onboarding-default-browser-nightly-16", "randomizationUnit": "nimbus_id" }, "schemaVersion": "1.6.2", "userFacingName": "firefox-ios-homepage-onboarding-default-browser-experiment-m1", "referenceBranch": "control", "proposedDuration": 28, "isEnrollmentPaused": false, "proposedEnrollment": 7, "userFacingDescription": "Multifeature experiment for homescreen feature", "id": "firefox-ios-homepage-experiment-m1", "last_modified": 1643181627892 } ```

[heres-maria-sv-inactive]

We have finished testing. You can find the TestRail test runs here. We have found the following issues:

• Bug 1752146 NEW Users can not be enrolled or unenrolled from Nimbus rollouts if studies are disabled
• Bug 1750705 REOPENED Users can not change Nimbus rollout behaviors using fallback preferences if the desired behavior is the preference default
• SDK-455 (CLOSED in favor of SDK-464) Users are not enrolled in an Android rollout recipe with the isRollout flag set to true

Desktop Version/Platforms Tested:

• Firefox Beta 97.0b3 through 8 (Build ID: 20220113185849 to 20220125201015) on Windows 10.
• Firefox Nightly 98.0a1 (Build ID: 20220116095124 to 20220126065618) on Linux Mint 20.2 set to beta channel.
• Firefox Nightly 97.0a1 (Build ID: 20211216094142) on macOS 11.6 set to beta channel.

Mobile version/Platform Tested:

• Android: Firefox Release null (built from the release_v96.0.0 branch as release build) (AS v86.2.1) (Build #2015858731, dcb09fcdc+) (AC: 86.2.1, 48e94ac636) (GV: 96.0-20220103221059) on Android 9 - Samsung Galaxy S8.
• iOS: Firefox Daylight 40.0 (Fennec) (AS: v.86.1.0) on the Xcode Simulator (iOS 15, iPhone 13 Pro)

[jaredlockhart]

Amazing thank you I'll wait for https://bugzilla.mozilla.org/show_bug.cgi?id=1752146 to be resolved and then move this forward.

[heres-maria-sv-inactive]

After the clarification of an expected behavior, this issue was also filed: EXP-2156 NEW Mobile Nimbus rollouts are not affected by the studies toggle

[jaredlockhart]

Okay blocking on https://mozilla-hub.atlassian.net/browse/EXP-2117 and then we can land the Experimenter parts

[jaredlockhart]

Thank you @heres-maria for the thorough testing!

[eliserichards]

Merged the schema change, but have not yet made a release. We have some sample data candidates here that I'd like to wait for before the release:

@jaredlockhart trying to wade through these comments. Is this issue good to close now, or do we still need to wait? Can this be released (I'm seeing the last release was 1.7.0 back in December)?

[eliserichards]

The new version v1.8.0 is being consumed by experimenter now. Closing this 👍