Engineering Review Checklist for Firefox WebExtension Experiments

[biancadanforth]

Engineering Review Checklist for Firefox WebExtension Experiments

This is a working checklist for reviewing WebExtension experiments in Firefox. Items may be added, edited or removed at any time.

• [ ] README:
  • [ ] Explains clearly what the extension does for each experimental branch and control.
  • [ ] Lists any extension configuration settings useful for testing.
  • [ ] Links out to a META bug and/or other documents with more context, including what Firefox channel and version the experiment is targeted for.
• [ ] manifest.json contains the following fields:
  • [ ] name
  • [ ] version
  • [ ] author
  • [ ] description
  • [ ] homepage_url
  • [ ] incognito (recommended value: "not_allowed")
  • [ ] hidden (value: true)
  • [ ] applications.gecko.id (typically of the form "study-name@mozilla.org")
  • [ ] applications.gecko.strict_min_version (preferable)
  • [ ] applications.gecko.strict_max_version (preferable)
• [ ] Extension follows guidelines for modifying Firefox Prefs.
• [ ] WE API engineers confirm that the extension's use of WE APIs (public, privileged or experimental) will not trigger known bugs.
• [ ] Extension does not cause feature regressions in Try server tests.
• [ ] Extension does not cause significant[1] performance regressions.
• [ ] Only source files used by the extension are included in the XPI.
• [ ] Additional reviews (e.g. data, legal, privacy, security, ...) may be required if:
  • [ ] The extension makes use of third party libraries and/or services.
  • [ ] The extension modifies web page content in any way.
  • [ ] Data is transmitted outside of Mozilla's telemetry pipeline.

[1]: As a reference, for a production application (i.e. not an experiment), a 2-3% worst case scenario regression in page load time would generally be considered acceptable. Since experiments are temporary and typically only go out to a very small percentage of users, we can often be more lenient.

---

Changelog for Engineering Checklist

[12/17/19]

• manifest.json contains the following fields:
  • Remove icons ("48" and "96")
  • Addon studies, when properly hidden, are no longer displayed in about:addons or about:debugging.
  • Add incognito key
  • This determines how the extension behaves in private browsing windows.
  • Add hidden key
  • This hides the addon in about:addons and about:debugging after it is signed.
• Add "Extension follows guidelines for modifying Firefox Prefs." list item.

[biancadanforth]

What the extension needs to handle

I will help you with much of this, but we should keep it in mind.

• Encrypt and package telemetry pings.
  • TBD if this or some other modification is needed. Pending Pioneer v2 specification.
• Check that the user is in the Pioneer cohort if using Pioneer for recruitment.
• Record study-specific telemetry.
• Implement user consent UI.
• Modify behavior in different privacy contexts as needed.

Open questions

• What channel and version of Firefox is this shipping in?
• What are the eligibility requirements for a user to participate in the study?
  • E.g. The user is in an en-us locale, part of the Pioneer cohort, ... Most of these can be handled upstream by Normandy, but some may require the experiment code to check.
• From our 11/27 e-mail discussion, the preferred deployment mechanism is to use Normandy with Pioneer. Is our Pioneer cohort large enough to draw from for this study?
• Is there a survey?
  • If so, how is that going to be integrated with the experiment?
• What telemetry do you want to record?
  • The Normandy client already records some general telemetry.
• Where can users learn more about the study?
  • The Options page will not be accessible via about:addons if the study is marked as hidden. One option is a SUMO (support.mozilla.org) page referenced in the study's description in about:studies.
• How is the extension supposed to behave in different privacy contexts?

Recommendations

• Read through Contributing to the Mozilla code base.
• Create a METRICS.md (example) document for any telemetry you want to record.
  • Get data collection approval.
• Work with the Mozilla project owner (rweiss or rhelmer) to create a tracking ticket for this study (likely in Experimenter or Bugzilla).
• Get UX and policy approval of user consent UI and user flow.

---

Changelog for Additional considerations

[12/17/19]:

• What the extension needs to handle
  • Add "TBD if this or some other modification is needed. Pending Pioneer v2 specification." under "Encrypt and package telemetry pings.".
  • Replace "Check that the user is in the Pioneer cohort." with "Check that the user is in the Pioneer cohort if using Pioneer for recruitment."
  • Add "Implement user consent UI."
  • Add "Modify behavior in different privacy contexts as needed."
• Open questions
  • Add "Where can users learn more about the study?
  • The Options page will not be accessible via about:addons if the study is marked as hidden. One option is a SUMO (support.mozilla.org) page referenced in the study's description in about:studies."
  • Add "How is the extension supposed to behave in different privacy contexts?"
• Recommendations
  • Add "Get data collection approval."
  • Add "Get UX and policy approval of user consent UI and user flow."