Create dashboard page comparing Fission vs non-Fission perf test results

[cpeterso]

I'm the EPM for the Fission team. We'd like to request a new dashboard page like https://health.graphics/windows/64 but comparing Fission vs non-Fission perf test results.

Tests

We have test results in Perfherder for the following test suites running on mozilla-central:

• Talos
  • "fission" tests enabled in bug 1594631
  • example Perfherder query for Talos tests labeled fission 📈
• Raptor:
  • enabled in bug 1594631
  • example Perfherder query for Raptor tests labeled fission 📈
• AWSY test results
  • "awsy-tp6-fis" was enabled for Fission in bug 1599621.
  • example Perfherder query for AWSY tests labeled fission 📈

I expect we'll enable Fission for more tests over the next 3-9 months.

Platforms

The Fission tests currently run on the windows10-64-shippable-qr and linux64-shippable-qr platforms, but we would only need to show the test results from Windows. (qr is "Quantum Render" aka WebRender. Fission requires WebRender.)

Do you recommend:

• creating a new page just for Fission vs non-Fission (something like https://health.graphics/fission)
• or adding the Fission results to the existing but crowded graphs on https://health.graphics/windows/64?

[klahnakoski]

I would prefer a new /fission page, but it really depends on what you are looking for. I would like the dashboards be specific to the project and the team that uses them. The current charts are crowded because there are multiple users with slightly different requirements, over a number of projects.

Are you comparing Fission another product? Chrome? geckoview? If so, then many of the charts on the current dashboard may had Fission added as yet-another-platform automatically.

[cpeterso]

I would prefer a new /fission page, but it really depends on what you are looking for. I would like the dashboards be specific to the project and the team that uses them. The current charts are crowded because there are multiple users with slightly different requirements, over a number of projects.

That makes sense. The https://health.graphics/windows/64 page has some tests and platforms that we don't need to monitor for Fission.

Are you comparing Fission another product? Chrome? geckoview? If so, then many of the charts on the current dashboard may had Fission added as yet-another-platform automatically.

The Fission team only cares about comparing Firefox with Fission-enabled vs Firefox with Fission-disabled. For 2020, we only care about Fission performance on desktop (so probably just Win64 is good enough).

[davehunt]

Note that the fission metadata has been standardised as fission. I will update the examples in the issue description.

[klahnakoski]

@davehunt I got an email from @cpeterso: It appears this work is now needed.

[klahnakoski]

@airimovici If this is done in health graphics, then I would imagine copying the TP6 desktop page (https://health.graphics/windows/tp6?test=warm-loadtime&platform=win64)

• copy /windows/TP6.jsx -> /fission/index.jsx
• a new route to point to /fission/index.jsx
• remove the test selector
• instead of groupBy('site') we would groupby('suite')
• make a new TP6_COMBOS (with better name of course) to hold all the suites.

[airimovici]

@klahnakoski For adding the fission vs non-fission charts to AWFY we need to define the charts in this file from the firefox-performance-dashboards repo: src/awfy.js https://github.com/mozilla-frontend-infra/firefox-performance-dashboards/blob/master/src/awfy.js

The current implementation from AWFY is too rigid for defining charts that compare by the extraOptions (we need that for comparing fission / non-fission) and it needs refactoring on a 2300+ lines of code file.

Looking at the Health Dashboards approach, it is a lot more flexible and it would require less effort tot do it like that.

[cpeterso]

Creating a separate Health Dashboard page sounds good to me. @davehunt said this dashboard request would be added to the backlog for the Fx Performance team's April sprint.

[cpeterso]

Jira ticket FXP-278: https://jira.mozilla.com/browse/FXP-278

[cpeterso]

In addition to graphing the Fission vs non-Fission performance benchmark results, we would like graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly users (like the "Performance Metrics" graphs on https://health.graphics/windows/64).

I've consolidated the Fission team's dashboard requirements into the Google Doc linked below. Editing and commenting on the requirements (and maintaining our edit history) will be easier in Google Docs that in this GitHub issue or the Jira ticket.

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit#

[davehunt]

In addition to graphing the Fission vs non-Fission performance benchmark results, we would like graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly users (like the "Performance Metrics" graphs on https://health.graphics/windows/64).

It may make sense to split this work out into a separate issue. I don't believe @airimovici has worked on the telemetry graphs, and it may be that @klahnakoski is more suited to take on that work.

I've consolidated the Fission team's dashboard requirements into the Google Doc linked below. Editing and commenting on the requirements (and maintaining our edit history) will be easier in Google Docs that in this GitHub issue or the Jira ticket.

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit#

Thanks!

[airimovici]

@davehunt I created another issue for the graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly users: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues/680

I started today this issue (this current one) and I'll keep in touch with @klahnakoski

[jeanygong]

In addition to graphing the Fission vs non-Fission performance benchmark results, we would like graphs comparing Fission vs non-Fission performance telemetry for Firefox Nightly users (like the "Performance Metrics" graphs on https://health.graphics/windows/64).

I've consolidated the Fission team's dashboard requirements into the Google Doc linked below. Editing and commenting on the requirements (and maintaining our edit history) will be easier in Google Docs that in this GitHub issue or the Jira ticket.

https://docs.google.com/document/d/1L1eefmSW4YxFLYcsjVpDfE6F0n9mVsfdrAAsEbMqOMw/edit# @klahnakoski Would you be able to tackle the telemetry graphs as per @cpeterso. Would this be something you can do in April timeframe?

[klahnakoski]

https://github.com/mozilla-frontend-infra/firefox-health-dashboard/pull/682

[airimovici]

I merged the PR for this issue: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/pull/682

The issue that is still open is for the performance metrics graphs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues/680

Example from Windows page:

[cpeterso]

I merged the PR for this issue: #682

Thanks! The page looks great! Can we close this Fission benchmarks issue now that we're tracking the Fission performance metrics (telemetry) in #680?

https://health.graphics/fission?platform=win64-qr