Schema Change for Adding Multifeature

[jaredlockhart]

Welcome Change Captain! ⛵️

When making schema changes, follow this guidance to avoid breaking changes:

• Rather than modifying an existing field, consider adding a new field with a new name and later deprecating the old field when it is no longer used

  • When modifying a field by adding and deprecating, document:
  • The expected behaviour of the old field in terms of the new
  • The expected behaviour when both fields are present. (e.g. in the presence of both fields, the new old field should be ignored)

• When deprecating an existing field, consider making it optional/nullable rather than removing it altogether and then removing it entirely when all clients/active experiments no longer depend on it

Proposing Schema Changes

• [x] Assign this issue to yourself
• [x] Add a comment below describing the changes at a high level, including all new/deprecated fields, their types, and their uses
  • [ ] Link to any related proposal documentation
• [x] Create a PR which introduces the schema changes and includes the following
  • [x] Changes to relevant types
  • [x] Changes to their associated tests
  • [x] New sample data demonstrating their use for each of desktop/mobile
• [ ] Send an email to the Services RFCs Mailing List with a link to the PR requesting feedback
• [x] Receive feedback, update the PR as necessary, and commit it
• [x] Create a release including the new changes
• [x] File tickets/epics for all Nimbus clients to implement the changes
  • [x] Nimbus Desktop
  • [x] Nimbus Mobile
  • [x] Add all filed tickets/epics in a comment below
• [ ] For deprecated fields, file a ticket in Nimbus Engineering to add a warning to the Nimbus UI that the field will be deprecated and provide guidance about how experiment/feature owners should migrate away from their dependence on the deprecated field

When Client Changes Are Complete

• [x] File tickets/epics for all Nimbus Clients for the new changes to be QAed by supplying the new sample data to the relevant (desktop/mobile) client(s) and verifying the intended behaviour
  • [x] Nimbus Desktop
  • [x] Nimbus Mobile
  • [x] Add all filed tickets/epics in a comment below

When Client QA is Complete

• [x] Add a comment below with a QA report that includes
  • [x] Which version of each client application was tested
  • [x] Link to any bugs which were detected in testing

When Bugs Are Resolved and Clients Are Released

• [x] File tickets/epics in Nimbus Engineering to implement the changes in Experimenter and add them in a comment below, including:

  • [x] Changes to relevant APIs
  • [x] Changes to UI
  • Note for deprecated fields
  • [x] First prevent the usage of deprecated fields in experiments targeting versions where those fields are no longer supported
  • [ ] Then make sure no currently active experiments still depend on them before removing them from UI/APIs
  • Leave the fields in the database so that historical data is preserved and the change can be easily reverted if necessary

• [x] Comment below indicating which version of each client application includes the changes

  • Hint: this is much easier to do by noting the last version of the client application that didn't have the change at the time of making the change

• [ ] File tickets/epics in Jetstream to adapt it to any incoming changes that affect its analysis

  • Note changing Jetstream and Experimenter may cause conflicts so try to coordinate both changes to go out as close together as possible

When Experimenter Changes Are Complete

• [ ] You're done! Close this ticket.

[jaredlockhart]

This change will introduce the ability to target multiple features with a single experiment. To support this, we are adding a new key to the Branch type called features which will contain an array of objects each of which contains

• featureId: string
• enabled: bool
• value: object

Deprecating the feature key on Branch will follow with a subsequent schema change.

[jaredlockhart]

Desktop/Experimenter Epic: https://mozilla-hub.atlassian.net/browse/EXP-1610

Mobile Epic: https://mozilla-hub.atlassian.net/browse/SDK-342

[heres-maria-sv-inactive]

We have tested the Multifeature client side as part of our weekly Nimbus testing in week Nov 1 - Nov 5 for mobile and week Nov 8 - Nov 11 for desktop. We have found two issues:

• SDK-447, Expose events experiments are indistinguishable from each other on Android
• SDK-440 (WON'T FIX), Users can be enrolled in multiple experiments of the same feature using the new schema

Desktop:

• Initial verification: Firefox Nightly 94.0a1 (Build ID: 20210913095054) on Windows 10, macOS 11.1 and Linux Mint 20 as part of Bug 1726190.
• Additional scenarios: Firefox Nightly 96.0a1 en-US (Build ID: 20211111211702) on Windows 10 as part of the Full Functional Test Run.

Mobile:

• Android: Firefox Nightly 96.0a1 (AS v86.0.1) (Build #2015843931) (AC: 96.0.20211104143116, 96ae23b628) (GV: 96.0a1-20211104094642) as part of the Multifeature Android Client Side test run.
• Android: Firefox Nightly 95.0a1 (AS v86.0.1) (Build #2015843195) (AC: 95.0.20211031143125, b0d8dddde7) (GV: 95.0a1-20211031095403) as exploratory testing.
• iOS: Firefox Daylight 0.0.1 (Fennec) built on Nov 4 for the Xcode Simulator as exploratory testing.

[heres-maria-sv-inactive]

We have performed additional testing. You can find the TestRail test runs here. We have found the following issue, which we understand was resolved after Bug 1738286 was implemented:

• Bug 1745182 DUPLICATE The upgradeDialog feature is not disabled in the Control branch of a Nimbus experiment

Desktop Version/Platforms Tested:

• Firefox Release 95 (Build ID: 20211129150630) on Windows 10, Linux Mint 20.1 and macOS 11.6.

Mobile version/Platform Tested:

• Android: Firefox Beta null (built from the release_v96.0.0 branch as Beta build) (AS v86.2.1) (Build #2015850483, 3c5b9f496+) (AC: 96.0.0, ea5bd2687c) (GV: 96.0-20211206194555) on Android 9 - Samsung Galaxy S8.
• iOS: Firefox Daylight 39.0 (Fennec) on the Xcode Simulator (iOS 15, iPhone 13 Pro)