Schema Change for Per-Experiment Validation Opt-Out

[brennie]

Welcome Change Captain! ⛵️

When making schema changes, follow this guidance to avoid breaking changes:

• Rather than modifying an existing field, consider adding a new field with a new name and later deprecating the old field when it is no longer used

  • When modifying a field by adding and deprecating, document:
  • The expected behaviour of the old field in terms of the new
  • The expected behaviour when both fields are present. (e.g. in the presence of both fields, the new old field should be ignored)

• When deprecating an existing field, consider making it optional/nullable rather than removing it altogether and then removing it entirely when all clients/active experiments no longer depend on it

Proposing Schema Changes

• [X] Assign this issue to yourself
• [x] Add a comment below describing the changes at a high level, including all new/deprecated fields, their types, and their uses
  • [x] Link to any related proposal documentation
• [x] Create a PR which introduces the schema changes and includes the following
  • [x] Changes to relevant types
  • [x] Changes to their associated tests
  • [x] New sample data demonstrating their use for each of desktop/mobile
• [x] Send an email to the Services RFCs Mailing List with a link to the PR requesting feedback
• [x] Receive feedback, update the PR as necessary, and commit it
• [x] Create a release including the new changes
• [x] File tickets/epics for all Nimbus clients to implement the changes
  • [x] Nimbus Desktop
  • [x] ~Nimbus Mobile~
  • [x] Add all filed tickets/epics in a comment below
• [x] ~For deprecated fields, file a ticket in Nimbus Engineering to add a warning to the Nimbus UI that the field will be deprecated and provide guidance about how experiment/feature owners should migrate away from their dependence on the deprecated field~

When Client Changes Are Complete

• [x] File tickets/epics for all Nimbus Clients for the new changes to be QAed by supplying the new sample data to the relevant (desktop/mobile) client(s) and verifying the intended behaviour
  • [x] Nimbus Desktop
  • [ ] ~Nimbus Mobile~
  • [ ] Add all filed tickets/epics in a comment below

When Client QA is Complete

• [x] Add a comment below with a QA report that includes
  • [x] Which version of each client application was tested
  • [ ] Link to any bugs which were detected in testing

When Bugs Are Resolved and Clients Are Released

• [x] Comment below indicating which version of each client application includes the changes

  • Hint: this is much easier to do by noting the last version of the client application that didn't have the change at the time of making the change

• [x] File tickets/epics in Nimbus Engineering to implement the changes in Experimenter and add them in a comment below, including:

  • [x] Changes to relevant APIs
  • [x] Changes to UI
  • Note for deprecated fields
  • [x] First prevent the usage of deprecated fields in experiments targeting versions where those fields are no longer supported
  • [x] Then make sure no currently active experiments still depend on them before removing them from UI/APIs
  • Leave the fields in the database so that historical data is preserved and the change can be easily reverted if necessary

• [ ] File tickets/epics in Jetstream to adapt it to any incoming changes that affect its analysis

  • Note changing Jetstream and Experimenter may cause conflicts so try to coordinate both changes to go out as close together as possible

When Experimenter Changes Are Complete

• [x] You're done! Close this ticket.

[brennie]

The intent is to add an additional field for experiments, validationOptOut, that, when true, would prevent desktop from performing schema validation for the experiment. This is intended for cases where an experiment needs to launch but is failing validation due to schema errors or problems in the validation logic and the fixes for those issues cannot ride the trains in time.

Proposal

[brennie]

PR https://github.com/mozilla/nimbus-shared/pull/213

[brennie]

Desktop bug

[brennie]

Experimenter console ticket

[brennie]

This landed in Firefox desktop 107.