Add graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly

[airimovici]

depends on https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues/640

In addition to graphing the Fission vs non-Fission performance benchmark results, we would like graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly users (like the "Performance Metrics" graphs on https://health.graphics/windows/64).

I've consolidated the Fission team's dashboard requirements into the Google Doc linked below. Editing and commenting on the requirements (and maintaining our edit history) will be easier in Google Docs that in this GitHub issue or the Jira ticket.

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit#

[jeanygong]

@klahnakoski can you assign yourself to this if you are taking on this work? I don't have access to to do this in github.

[klahnakoski]

@jeanygong i think this is done @airimovici if nothing is broken in your PR, please merge it.

[jeanygong]

@klahnakoski thanks for the update.

[cpeterso]

@klahnakoski , PR #682 creates a new dashboard page with TP6 and Talos performance test results, but we'd also like to graphs comparing Fission vs non-Fission users' telemetry for the telemetry probes listed in the doc below. Something like the "Performance Metrics" section on https://health.graphics/windows/64.

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit#heading=h.2r1cri2435pz

[klahnakoski]

@cpeterso Thank you, now I better understand: I did not realize there was a telemetry component. The doc helps plenty. We will require a pointer to where that telemetry data can be found.

[cpeterso]

@cpeterso Thank you, now I better understand: I did not realize there was a telemetry component. The doc helps plenty. We will require a pointer to where that telemetry data can be found.

Here is a list of the telemetry probe names we'd like to graph:

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit#heading=h.2r1cri2435pz

I'm not sure where the telemetry data is fetched from, but the code for the https://health.graphics/windows/64 page's "Performance Metrics" section:

https://github.com/mozilla-frontend-infra/firefox-health-dashboard/blob/90d92e7f9d21134df3b9e5fd63801e1819f646fd/src/windows/Windows.jsx#L311

... appears to delegate fetching to a TelemetryContainer component:

https://github.com/mozilla-frontend-infra/firefox-health-dashboard/blob/8af8ed56df47e9f681e2b73cd09a02501604d4ee/src/telemetry/graph.jsx#L10

Are the probes we care about for Fission available in whatever backend data source TelemetryContainer uses? If not, I can try to find out who manages the backend data source.

[cpeterso]

The Jira ticket for this feature request is in the Performance team's May sprint: https://jira.mozilla.com/browse/FXP-904

[cpeterso]

We can close this issue as WONTFIX. The Jira ticket https://jira.mozilla.com/browse/FXP-904 has been canceled.

IIUC, while comparing Fission vs non-Fission telemetry will be trivial in a proper A/B experiment, we can’t easily comparing Fission vs non-Fission telemetry for arbitrary probes (e.g. on telemetry.mozilla.org or health.graphics dashboards) because the telemetry data is not tagged as Fission in a queryable field.

Also, arbitrary comparisons between Fission vs non-Fission users is not meaningful because we have very few Fission users (only a couple hundred) and they are all opt-in.