Create mach-test.sh script for local Firefox development

[biancadanforth]

Note: This method involves installing the addon in a local Firefox build as a system addon as opposed to a temporary addon as done in npm run firefox. One major difference in testing is that the initial reason code passed into Feature.jsm is ADDON_STARTUP instead of ADDON_INSTALL as in npm run firefox (our current method of testing the addon via Selenium WebDriver). Since the intro panel for this study is only shown on install, it doesn't show up when running the addon on the local build of Firefox using ./mach run.

Based on this, though jar.mn and moz.build were largely modeled after files from the shield-recipe-client addon to get it to work. The shield-recipe-client is also a system addon, and its source code can be found on DXR/SearchFox and in a local build of Firefox in ./browser/extensions.

Modified some JSMs such that the addon can run with both npm run firefox and ./mach build/ ./mach run.

Added a build-includes.txt file (modified from here) and a mach-test.sh (modified from here) that puts a folder (folder name is the addon name, i.e. tracking-protection-messaging-study) into the ./dist folder. This file can then be directly copied into ~/src/mozilla-unified/browser/extensions and, along with a few other steps, can be used to test the addon on a local build of Firefox via ./mach build, ./mach run.

./mach test empty works. Just need to find out what tests to run!

TODO

• Update ShieldUtils.jsm build step so that it is added to ./addon/lib instead of ./addon.
• Consider ways to further optimize testing on a local build of Firefox, including using a symlink to link the output folder from mach-test.sh to the correct location in the Firefox directory for system addons (~/src/mozilla-unified/browser/extensions).

[biancadanforth]

Draft process thus far:

Alternate resource

How to Build/Run/Test Your Shield Addon Using Firefox Test Suites

Motivation

This is an alternative to maintaining/developing our own automated testing environment via Selenium WebDriver. If we use the test suites native to Firefox, there is a lot of example code for writing the tests, and a lot of additional tests that can be run through the Try server including performance and TALOS tests. We would have the benefit of leveraging custom CI/test runners maintained by Mozilla. Advantages:

• We don't have to maintain/update our own automated tests/CI/test runners
• By using MochiTest/XPCshell and other Firefox tests, we can look at example tests in searchfox or dxr and consult other Firefox developers on testing.
• There are additional tests that will check for regressions, performance impact, etc. that we would get for free.

./mach build and ./mach run

• Note: This process installs your addon in the tree as though it were a SYSTEM addon, like the Shield Recipe Client, unlike the Shield template’s Selenium WebDriver script npm run firefox, which installs the addon as a temporary addon. Here's rhelmer's patch that covers the tasks we will be walking through (note: the jar.mn in this patch does not work; we will use a different file as a reference).
• Note: since this install location is locked, you can't install updates to it, updates happen in a different location that supersedes this one (the one in the profile)
• Note: This process currently has you manually copying over files, say from a GitHub repo into the Firefox source code. As a result, this is not the recommended method for general addon development in tree, but there could be optimizations, such as using a symlink for your GitHub repo directory files inside your local Firefox build (per rhelmer).
• Note: ./mach build, even an artifact build, for me has taken anywhere from 30 seconds to 30 minutes. So this path is not the best for rapid prototyping of an addon. Could possibly be used for testing in tree at the end of development. The 30 minute build time was after not updating my local copy of Firefox for a few days. The more frequently you update, the less the build time.
• Note:
  • We need to ensure one of the two experimental branches ("fast" or "private" i.e. worst case scenarios) are what are tested, particularly for performance.
  • Test infrastructure prohibits outbound connections (e.g. end-of-study surveys), as those can prevent perf tests from being repeatable and can be exploitable.

1. Create a moz.build and jar.mn file for the addon inside its root directory
   • To ensure you set up the jar.mn correctly, put this in the address bar resource://shield-recipe-client-content/AboutPages.jsm, then look in shield-recipe-client's jar.mn and content dir, you can see how it's wired up, and you can test to see where your own JSM files are that way too.
2. In the terminal, navigate to ~/src/mozilla-unified and hg pull and hg up to get the latest copy of Firefox.
3. In Sublime, edit the ./browser/extensions/moz.build file to add the addon (by its ID name, see the install.rdf file, ex: tracking-protection-messaging-study@shield.mozilla.org would enter “tracking-protection-messaging-study” as its name) to the list of system addons.
4. In the File Explorer, manually put a copy of the addon file (as depicted above) into ~/src/mozilla-unified/browser/extensions, with the top level folder’s name equal to the addon’s name (ex: “tracking-protection-messaging-study”). Note: the ./bin/mach-test.sh script from this PR will create a folder with all the files you need inside of the ./dist directory
   • Add a test subfolder to include your own linting file and tests as desired.
   • Ensure the file structure is similar to that shown for “shield-recipe-client”, i.e. don’t have Config.jsm or StudyUtils.jsm in the root of the folder as those won’t load properly. You must put them in ./content or ./lib for example. The only files in the root folder for the addon should be:
   • bootstrap.js
   • install.rdf
   • jar.mn
   • moz.build

Also, ensure the paths to Config.jsm and StudyUtils.jsm are resource:// (e.g. they do not use SCRIPT_URI_SPEC). A good example for a URL for Config.jsm is: “resource://tracking-protection-messaging-study/lib/Config.jsm”.

• Note: There’s may be a way to configure the jar.mn file to read other paths than subfolders inside resource URLs.
• Also, ensure the install.rdf file is accurate, particularly the addonID is consistent with what you are using to reference the addon in step 2, in the jar.mn file and in the resource:// URLs in your JSMs. Also, if your addon is NOT an embedded webExtension, make sure the corresponding tag in the install.rdf file is set to “false”.
  5. Now you are ready to try to build and test the addon. Type hg status to see all the changes you have made to ensure they seem right. 5.1 If they do, type hg add . in the terminal. Typing hg diff will show you what changes you made to each file, including new files that were added.
  6. Build Firefox with ./mach build After a few seconds, you should get a message “Your build was successful!”; you may have to troubleshoot if there were any issues. I only had a one or two minor issues related to install.rdf, mentioned above.
• A local build logs both to the terminal and to the browser console.
  7. Go for a test run with your local copy of Firefox with ./mach run.
• Note: If you were previously testing your addon with Shield’s npm run firefox script (loads the addon as a temporary addon), you may need to make some changes to file paths for your resources. Converting all my files to be located in subfolders in my addon directory and mapping them all to resource:// URLs fixed my problems. You can verify that the files you need are at the right location by entering the resource:// URL into the URL bar in Firefox after ./mach run. You should get the raw text for the script output to the page.

./mach try/test

These instructions are for testing against "mozilla-central". For testing against the "release" channel, see this. Good resource: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing

Once your patch is at the tip of your branch of interest (ex: the tip of central) and it is also the "current commit" (this can be checked with hg wip, and the patch can be made the "current commit" with hg up <commit-hash>), and you have successfully built it (./mach build) and run it locally (./mach run), you are ready to test it!

Note: It's a good idea to check one final time before you ./mach build and ./mach run your patch to check hg export .. This will show you the full patch, including the full commit message and diff. This way you can ensure the changes you made are reflected in the patch you push to try.

There are two ways to test your patch, locally on your machine and on the try server.

Note: There is a temporary profile used for ./mach run that is cleared out every time you ./mach clobber. ./mach run creates a new profile if it does not exist, and it goes into ./${OBJDIR}/tmp/scratch_user/. Local testing and try server testing use a new profile each time.

Local testing

If it has been a while (even 1 day in my experience) since you built Firefox locally on your machine, you may need to run ./mach build (faster) before you can run tests.

Note: ./mach build faster doesn't rebuild any of the compiled (C/C++/Rust) code, just JavaScript/HTML/CSS/XUL.

Note: You can set debugger; breakpoints, for example, in the tests themselves.Though it's a lot more annoying with debug builds - much better with opt, use the Browser Toolbox debugger. You can spawn one and hold your test from running with:

/mach mochitest <path to test> --jsdebugger

That'll open up the browser, the browser toolbox at the debugger, and another window to trigger the test run from.

See Debugging failed tests tips and tricks for more suggestions on local testing.

Unit tests

Unit tests produce a binary result: pass or fail. These are the easiest kinds of tests to run locally.

To run a single unit test, you can use the following command: ./mach test <path/to/test>

E.g. ./mach test dom/tests/browser/browser_noopener.js

This is particularly helpful if you'd like to re-run a test that failed on the Try server after you've made changes.

Note that in the test summary, you will see a number next to "Passed" and "Failed", these correspond to a particular check (ex: Assert two things are equal) in the test. So if the total number of Passed and Failed items is 5, that means the test checks 5 things. You can see the test log to find which of these checks your patch failed at.

Note: reftestcount logging is not enabled by default, so if you are trying to test for memory leaks, for example, the local test will not be able to detect this kind of failure without editing mozconfig. I had trouble getting this to work, so I've kept my memory leak testing in the try server for now.

How to view the results: The results of the test should be output to your terminal.

Perf tests

In contrast to unit tests, perf tests are not pass/fail. They collect much more data as a part of the test and attempt to plot that data, comparing it to the branch of interest (e.g. central) without the patch. Looking at the results of a perf test run locally is a little more challenging as a result.

To run a single perf test, you can use the following command: ./mach talos-test <args>

E.g. ./mach talos-test -a tp5o

How to view the results: TODO ask jmaher?

Try server testing

This uses remote machines to perform the testing. Before you can run tests on the try server, you need Level 1 commit access.

You can trigger a try run from the command line like so: ./mach try <try syntax>

The art here is knowing which tests are important to run, as running all tests is very costly, time consuming and unnecessary. There is a webpage that helps you choose try syntax based on the tests you want performed.

For my first shield study test, I did: ./mach try -p linux64,win64,macosx64 -u all -t all --artifact This runs all unit tests, all perf tests (TALOS) on all desktop platform on an artifact build.

The terminal will log a bunch of links for your reference, including a link to the results of all your tests in Treeherder. You will also receive an e-mail with this information.

After you start the tests, you should keep an eye on the Treeherder page and watch for any problems. In particular, watch very early on for red Bs, this means the build has failed (and therefore that particular platform/build will not be able to continue its tests). If most of your builds are failing, that might be a sign you should cancel the test run completely and dig deeper. If only one or two are failing, but some are building, particularly for the same platform, then try rerunning the build job to see if the build completes.

Here is a helpful resource that explains what the abbreviations for each type of test being run are. For example, a common problem is that the Build step for each test fails, meaning that it doesn't even get to perform that test before exiting. In cases like this, it's important to cancel the build as quickly as possible to avoid wasting testing farm resources.

Viewing test results on Treeherder and Perfherder

Perfherder To view a performance test in Perfherder, select a performance test from the list of tests displayed in Treeherder (ex: tp5). A panel will open up at the bottom of the page. Navigate to the 'Performance' tab, and click "Compare results against another revision". This will open a new window, where you can select a baseline to compare your patch's performance against.

After selecting your baseline, a new page will load with all the test results. At the top there are some filtering options. Select "Show only important changes" and "Hide uncertain results". This will show you performance regressions of particular interest from your patch.

[biancadanforth]

Switching to another branch temporarily; note to self to modify Config.jsm for any tests to force the "fast" experimental branch and remove study ending external links. out.txt

[biancadanforth]

How to test your Shield study against the release channel (instead of mozilla-central)

Many Shield studies are not intended to ship in Nightly; in fact, they often are intended for the "release" version of Firefox. It's recommended to test your study in the intended channel/version of Firefox. The process for testing against release is a little different than testing against mozilla-central, in part, because release is not where on-train AKA the vast majority of all Firefox development takes place.

Resource: https://github.com/biancadanforth/tracking-protection-shield-study/issues/26#issuecomment-373173337

Note: The helper bash script used here is from a script I wrote for the Tracking Protection Messaging Shield study.

Preparations

Note: These instructions are based on the commands for OSX.

1. Set up your local build of Firefox.
2. Configure ~/src/mozilla-unified/mozconfig. You can find information about build configurations here. Here’s mine:

   # Automatically download and use compiled C++ components:
   # ac_add_options --enable-artifact-builds
   # When building artifact builds, the build will go into this directory
   # If need to do a full build, comment out artifact builds and MOZ_OBJDIR
   # This saves time when switching between artifact and full builds
   mk_add_options MOZ_OBJDIR=./objdir-frontend-debug-artifact
   # Debug builds are used to add additional logging, which is required for example to run unit tests locally to check for memory leaks
   # ac_add_options –enable-debug

   Important: At the time of writing this, there were not artifact build files available for the Release channel. I had to do full builds. This is the relevant bug.

3. Create your add-on’s jar.mn and moz.build files per these instructions in the ./addon folder of your project repo.

Editing some non-add-on files in the tree

1. Navigate to the source code directory. a. cd ~/src/mozilla-unified
2. Fetch the latest changes for your channel of interest and update the repo to the tip of that branch: a. hg pull -u release
3. Make the tip of the channel branch the “current commit”. a. hg up -r release b. Pro-tip: You can verify that your channel is at the tip of the repo and is the “current commit” with hg log -G. If the channel branch is at the top of the list, it’s the tip. If that commit has an @ symbol to the left of it, it’s the “current commit”.
4. Edit the artifacts.py file: ./python/mozbuild/mozbuild/artifacts.py#119 to add releases/mozilla-release to the list of CANDIDATE_TREES. This change must be a part of your patch a. By default, artifact builds do not download artifacts for release, since it’s not a common workflow.
5. Add your add-on’s Shield study ID to the list of DIRS in ./browser/extension/moz.build. This is the value before the@symbol in your add-on’sinstall.rdf`id element. a. Ex: <em:id>tracking-protection-messaging-study@shield.mozilla.org</em:id> would mean I would add “tracking-protection-messaging-study” to the list.

   DIRS += [
   'activity-stream',
   'aushelper',
   'followonsearch',
   'formautofill',
   'onboarding',
   'pdfjs',
   'pocket',
   'screenshots',
   'shield-recipe-client',
   'webcompat',
   'tracking-protection-messaging-study',
   ]

   

Adding your study and making the commit in hg

1. Open a new tab in the terminal.
2. Ensure your working directory state in your project repo is the state you want to push to Try. a. Note: You should force your worst case scenario treatment branch in terms of performance and complexity when testing on the Try server. b. Remove any external links from your study (common place for all studies are the surveys at the end of the study in Config.jsm; for this patch, just replace them with null).
3. Navigate to your add-on’s GitHub repo
4. in the terminal, run npm run test_on_try a. This will bundle all files in ./addon listed in the build-includes.txt file; currently, this omits LICENSE, chrome.manifest (jar.mn handles the resource:// mapping).
5. Switch back to the first tab with the hg mozilla-unified repo and check that the files were copied over successfully with hg status into the browser/extensions/<add-onID>/ folder. a. You should see, in addition to your add-on files, that ./python/mozbuild/mozbuild/artifacts.py and ./browser/extensions/moz-build have been modified. b. You should also have a jar.mn and moz.build file inside your ./addon folder in your Git repo that was copied over. For help making these files, see this post.
6. When everything looks good, make your commit via hg add . and hg commit -m “Commit message…” a. Note: If you make a mistake and need to modify a file in your code, you can switch back to the terminal tab with your project repo, make and save the change, and re-execute npm run test_on_try. Then you can ammend your commit with the modified files with hg add . and hg commit --amend. b. Optional: Add a bookmark to your commit with hg bookmark <bookmarkName>, this lets you reference the commit by the bookmark name rather than its commit hash. (You need to add the bookmark extension first to do this.) Keep the bookmark name short and memorable for easier referencing. You can see a list of your bookmarks with hg bookmarks.

7. Verify your commit is at the tip of the mozilla-unified repo just above the “release” tip, and that it is the “current commit” (the @ sign is next to it) with hg log -G (you can also use hg wip to see the tips of each branch).

   

   a. Note, if your commit gets buried after pulling updates from Release some time later, you can "rebase" your commit on top of the tip of Release with hg transplant <your commit hash or bookmark>. This moves your commit on top of release; note: transplant is an hg extension. It may need to be added to your hg config.

8. Build Release Firefox locally on your machine with your patch ./mach build a. If you haven’t built release recently (ex: you are coming from testing in central) you will likely have to do a full build instead of a ./mach build faster.
9. Make sure the addon seems to be working properly one last time, and the Firefox version corresponds to the current version for the release channel: ./mach run. a. Note: The shortcut icon will still show the Nightly logo. Branding configuration is handled separately, and can be changed by adding ac_add_options --enable-official-branding to your ./mozconfig file.
10. ./mach try <try syntax> a. Push to Try! b. Ex: ./mach try -p linux64,win64,macosx64 -u all -t all --artifact runs all unit tests, all perf tests (TALOS) on all desktop platform on an artifact build. c. Ex: ./mach try -p linux64 -u all -t none --artifact This will run all unit tests on linux desktop platforms with an artifact build. d. The terminal will log a bunch of links for your reference, including a link to the results of all your tests in Treeherder. You will also receive an e-mail with this information. e. Keep in mind that testing on macOS is extra costly as the number of physical machines is limited compared with Linux and Windows (as of 02/09/18).

[biancadanforth]

Debugging Failed Tests Tips and Tricks

Some of these tips will only apply to unit tests, but for the most part they're generally applicable. By "branch" I am referring to the correct Firefox channel, like mozilla-central (Nightly), inbound, release, etc.

1. Repeat the test several times to make sure it is not intermittent. Also check in Treeherder for the test to see if there are any intermittency bugs associated with the failure you are seeing, and if that could be the issue.
2. Run the same test against the tip of the correct branch without the patch to confirm that it passes consistently.
3. Look at the source code of the test. For non memory leak unit test failures, you can look at the test log to find where in the test the error is being thrown.
4. If you can't tell what the test is testing for or what the error means, go to dxr or searchfox and do a blame to find who edited the test last and ask them.
5. If you think you've found the problem, double check that the problem exists in your patch and not in the branch of interest.
6. If you're bisecting your code as part of troubleshooting, make sure you've gone as far down into the code as possible for where the test fails. Ex: if you stop at a listener, does that listener get called during the test? You may have to go into that listener. Also keep in mind when you find the exact line causing the failure, that the error could be due to that line in combination with something else executed elsewhere. I.e.: Is a shared object being modified? If so where else is that object being used.
7. Add breakpoints in your code and also in the test as needed to pinpoint the exact line. Look at the callstack in Debugger and make use of watch expressions. Keep in mind there could be more than one callstack. Ex maybe a function gets called at multiple times in the code.
   • If testing locally, you can add breakpoints in your code or the test code with debugger; and the --debugger option to your test command to automatically open the debugger before the test starts.
8. May the force be with you. As long as there is hope, all is not lost.