Performance Testing

[Standard8]

The basic summary of what to do is:

• Create a patch to insert the add-on into mozilla-central (see below)
• Do two try builds. The first is without the patch, the second is with the patch. They should be based off the same revision of mozilla-central. Use a try chooser command such as try: -b o -p linux64,linux64-pgo,macosx64,win32,win32-pgo,win64,win64-pgo -u none -t all --artifact --rebuild 6 (not you need at least 6 rebuilds for enough comparisons, you may need a few retriggers as well). You should also be able to get away with artifact builds.
• At some stage you'll get an option to build a link, if not you can use this link and just update the revisions for it.

Once the results come in, they'll appear on the comparison page, and you'll want to look for any red or yellow changes.

For building into mozilla-central. The easiest way is to copy what happens in one of the browser/extensions/ folders of the system add-ons - probably the screenshots one.

• Add a directory, and add it to browser/extensions/moz.build
• Copy the add-on files into that directory so that install.rdf/bootstrap.js are in that folder.
• Then take a look at screenshot's moz.build and adapt that for your files.
• You should then be able to build & run Firefox, and the add-on should be present.

If you need more help or have questions, feel free to ping me.

[jaredhirsch]

I can take this. I did all this stuff for Screenshots.

[jaredhirsch]

Base commit with Talos jobs: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0d75aa9246a904480470ec0eafab6fb7589d3d61

[jaredhirsch]

Talos run with min-vid shield study branch + moz.build file, no performance optimizations: https://treeherder.mozilla.org/#/jobs?repo=try&revision=efef136c524244832c1e962cf339cb41b8a6a068

Comparison between this and the base: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0d75aa9246a904480470ec0eafab6fb7589d3d61&newProject=try&newRevision=efef136c524244832c1e962cf339cb41b8a6a068&framework=1

[jaredhirsch]

Here's a Talos run with the sessionstore-windows-restored delay added: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d4a9f2c702f7103c5a567b1673907f36e7cc3c7

Comparison between this and the base: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0d75aa9246a904480470ec0eafab6fb7589d3d61&newProject=try&newRevision=2d4a9f2c702f7103c5a567b1673907f36e7cc3c7&framework=1

[jaredhirsch]

Oh no! I applied these patches on top of mozilla-central, but min-vid doesn't work in Nightly.

I'll have to figure out which commit / branch is the correct base for current release channel, then re-run the Talos tests there :-\

[jaredhirsch]

Re-running Talos tests on top of the FIREFOX_59_0_RELEASE tag. mach couldn't find artifacts for a commit this old, so I'm modifying the try incantation to remove the --artifact argument:

try: -b o -p linux64,linux64-pgo,macosx64,win32,win32-pgo,win64,win64-pgo -u none -t all --rebuild 6

Base commit: https://treeherder.mozilla.org/#/jobs?repo=try&revision=febd1a18cbce90f99f95f303c379ba0dff7abb64 Min-Vid without startup delay: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b80aff44b15dc684202645eb9f92b616bb691903 Min-Vid with startup delay: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a649cfd06b617538dcb3db023486ff41be5c8f7c

[jaredhirsch]

Hmm. Adding a startup delay does not seem to resolve the Talos perf issues:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=febd1a18cbce90f99f95f303c379ba0dff7abb64&newProject=try&newRevision=a649cfd06b617538dcb3db023486ff41be5c8f7c&framework=1&showOnlyImportant=1

[jaredhirsch]

Splitting the problem space to try to understand the source of the regressions:

• Reduce URL scanning in content scripts, by only checking the page's URL, not the URLs of embeds:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=757bee977b2ea4b662280d8463efe00a44822239
• Don't run content scripts at all:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=c5712f41e8a377ed0cbefd5f996ef0eed38e0a56

[jaredhirsch]

It seems like this might not get out before 60 goes to release on 5/9, so here's the same set of patches as Try runs against the release 60 base commit:

• Base run:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=ed1ceac6daf75a270cc96511e5dffbbf3a1c48ea
• Minvid:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=94ebd0d6413a2b463ebc9149b68c5467b41bbb90
• Minvid with startup delay:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=139cf0dfb9559f167f127ae2c2e4e8f87f60f321
• Minvid with content scripts not running in all_frames:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcbc797d0da297a57c7f7b74b3b9c7cdc4a4e49b
• Minvid without content scripts:
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=7049c63f0c3f897444aaf0649772f547d1978c35

[jaredhirsch]

Still seeing the huge tp5o regressions with content scripts totally disabled in 59 (content_scripts section removed from the manifest):

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=febd1a18cbce90f99f95f303c379ba0dff7abb64&newProject=try&newRevision=c5712f41e8a377ed0cbefd5f996ef0eed38e0a56&framework=1&showOnlyImportant=1

[jaredhirsch]

OK, starting to narrow down a little.

Something either in the StudyUtils.jsm code, or in our use of that code, seems to be causing a lot of the performance problems.

Here's a screenshot of the tp5o regressions on Firefox 59 + minvid + startup delay (results page):

If the code that loads and uses StudyUtils.jsm is removed, there are fewer reds, but still a lot of oranges ([results page]()):

More detail below, for later reference.

---

Today, I kicked off Talos runs testing three different changes on both minvid+59 and minvid+60:

1. effectively disable the whole addon: make startup() and shutdown() do nothing inside bootstrap.js
2. disable most of the webextension code: remove the background and content_scripts entries from the webextension manifest (so that no code is loaded into the background page & no URL filtering happens)
3. just remove StudyUtils-related code (partly because I have no idea what this code does; it's minified)

Here are the corresponding Try runs:

minvid-on-59

Talos run against just 59, no minvid: https://treeherder.mozilla.org/#/jobs?repo=try&revision=febd1a18cbce90f99f95f303c379ba0dff7abb64

1. startup/shutdown no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=523621d888f1d3b7f4c892a4958c1b1e9a199b3d
2. webextension no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=80c41a123dd41ee6d7d8ebf531ddb85fdf00405a
3. StudyUtils removed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fff2670382ac98add5a04e264c64eea6fca9c6b

minvid-on-60

Talos run against just 60, no minvid: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ed1ceac6daf75a270cc96511e5dffbbf3a1c48ea

1. startup/shutdown no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=05db4e59614ce6254e46d6cb30e0f4244102efc9
2. webextension no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=80c41a123dd41ee6d7d8ebf531ddb85fdf00405a
3. StudyUtils removed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8529a0c5b38a04315c83c8bdace17be9d73cc424

[jaredhirsch]

Here's a new run, with StudyUtils removed and where the embedded webextension is never started:

on 59: https://treeherder.mozilla.org/#/jobs?repo=try&revision=acd21af2719779fac4ab04a0d50cc1a0f921d597

on 60: https://treeherder.mozilla.org/#/jobs?repo=try&revision=46c310a5fa60b1df3efb171bd68f1e1cd23be608

[jaredhirsch]

The last run against 60 produced some inexplicable performance improvements, so I'm re-running everything now:

• base commit
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=7fa3fc7d9eaf7ff7db8a70ea6fa012f859d1a47d
• minvid, no startup delay
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc8e7046638ce89425833611c08227c19851c8e5
• minvid with startup delay
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=b617a4211cdf5745b73c9a5c4bdb75ca261da0df
• minvid with startup delay + remove StudyUtils code
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=f3fc6daf6950f8d83358a00b598e73b7a1dfb74c
• minvid with startup delay + remove StudyUtils code + don’t start the webextension at all
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=0240cc9b631805e095e3b4a2787b623b76c3c256
• new, suggested by mossop: minvid with startup delay + remove the onUpdated listener for the onboarding modal
  • https://treeherder.mozilla.org/#/jobs?repo=try&revision=8408f8426879e0a2d7dc7805dfe3494b17b93f67