Docs needed on how to run performance regression tests for study add-ons

mozilla / shield-studies-addon-utils

Mozilla Public License 2.0

6 stars 21 forks source link

Docs needed on how to run performance regression tests for study add-ons #269

Open motin opened 5 years ago

motin commented 5 years ago

As a study add-on engineer, I might worry that my add-on affects the performance of the client, but currently there is no documented way to verify this.

motin commented 5 years ago

@biancadanforth Iirc you ran performance regression tests for one of your studies. Did you document how yet?

motin commented 5 years ago

Ping @rhelmer

motin commented 5 years ago

Here is an example of a performance-related issue that may have been caught by a performance regression test: https://github.com/motin/jestr-pioneer-shield-study/issues/12

rhelmer commented 5 years ago

So @biancadanforth did this using the Mozilla TryServer: https://wiki.mozilla.org/Build:TryServer

At the time we had to wire it into the build system, but now it's possible to just drop a XPI into the right place and push and perf/unit tests should run.

motin commented 5 years ago

now it's possible to just drop a XPI into the right place and push and perf/unit tests should run.

Great news! As for docs, it should be enough if that wiki-page gets updated with this referenced right place. We can then link to the wiki page from the utils-docs.

biancadanforth commented 5 years ago

The last time I did it (earlier this year), I did write a lot of documentation, but I did not share it widely as @rhelmer and I agreed it was too complex to be shared out (leading to Bug 1451159).

Significant improvements to the process landed afterwards, including:

Bug 1451159 (drop XPI into a directory to run Mochitests)
Bug 1458571 (drop XPI into a directory to run TALOS tests)

I do not have a document for this new simplified process. Though most of it after dropping the XPI in the right directory should be the same as the process for general Firefox development.