TESTPLAN.md

[biancadanforth]

TESTPLAN.md

Preparations

How to obtain a Firefox build with the patch

Preferred method

1. For a given Bugzilla bug (ex: 1462725) that you are NI-ed on, under the "MozReview Requests" section, click on any of the links in the "Diff" column.

   

2. You will be taken to Review Board. In the second section of that page, notice the header "Commit(s) successfully landed on try". To the far right of that header are a couple of links. Click the "See it in Treeherder" link.

   

3. You will be taken to the Treeherder page for this patch. Select the desired OS opt build job (ex: for Mac OS, this would be "OS X Cross Compiled opt").

4. A new section will appear at the bottom of the page. In the "Job Details" tab, scroll down until you find the row with the Firefox binary for the chosen OS (for Mac OS, this would be the row containing "artifact uploaded: target.dmg"), and click on the link in that row. This will download the file.

Note: On a Mac, you will be prompted to move the "Nightly" application to the Applications folder. Please note that you can only have one instance of Firefox opened at a time, unless you open the second instance of Firefox from the "Terminal" application's command line (ex: /Applications/Nightly.app/Contents/MacOS/firefox-bin).

Alternate method

See README (#1 ) Development section.

How to find event telemetry pings in about:telemetry

1. Navigate to about:telemetry in the URL bar. By default, the "current ping" will be selected. Event telemetry is currently appended to the "main ping", though there are plans to change this in the future.

2. You can change which ping to show by clicking the down arrow next to the "current ping" label in the top left corner of the page.

   

3. Select the "Archived ping data" radio button and under "Ping Type", choose "main" in the dropdown menu.

4. You can choose between different main pings under "Ping" in the dropdown menu. For QA purposes, we will typically be looking at the "current ping", which is often the "main" ping, or we will be looking at the top two most recent "main" pings.

5. Clicking off that menu, note the sidebar of different kinds of data available in the main ping. If and only if there is any event telemetry recorded in a given "main" ping, there will be an "Events" tab in the sidebar.

   

Functional tests

Basic Study Behavior (Bug 1462725)

Study Preference

• When the shield.savant.enabled boolean pref is set to true in a clean profile through about:config, performing a search in the URL bar will generate a corresponding ping in about:telemetry. Conversely, when the shield.savant.enabled boolean pref is set to false in a clean profile through about:config, performing a search in the URL bar will NOT generate a corresponding ping in about:telemetry.

Eligibility

• If the user is in auto private browsing mode ("browser.privatebrowsing.autostart" set to true) and then the study is enabled ("shield.savant.enabled" set to true), the study will end ("shield.savant.enabled" set to false and "end_study" telemetry ping sent with an object value of "ineligible"). No further "savant" category probes will show up in about:telemetry.
• Set "browser.privatebrowsing.autostart" to false. Then set "shield.savant.enabled" to true. The study should start up, recording all “savant” category probes. Then set"browser.privatebrowsing.autostart" to true. Close out and restart Firefox. On reopening Firefox, the study should end with reason “ineligible”, submitting a final ”end_study” savant ping and no further “savant” category probes will show up in about:telemetry.

End study

There are only 3 study endings for this study (See the README for a description of each of these endings):

1. "ineligible" (see checks under "Eligibility" section)
2. "expired" (see checks under "Study duration" section)
3. "study-disable"

To check that the "study-disable" event fires correctly:

1. Open Firefox
2. In about:config, set the "shield.savant.enabled" pref to true.
3. Double click the pref once more to toggle it to false.
4. Notice in about:telemetry that there is a "savant" event telemetry ping with a method of "end_study" and an object of "study-disable".

Study duration

Test that when the study expires, it stops collecting data for the ”savant” probes (requires a try build, see Study Preference section above).

Study expired during the session.

1. Go to about:config and create a new int pref with a name of “shield.savant.duration_override” and a value of 10000.
2. Set shield.savant.enabled to true.
3. Perform a search in the about:newtab awesomebar.
4. Notice that the ”savant” newtab search event shows up in about:telemetry.
5. Wait about 30s (the timer is not very accurate).
6. Restart Firefox.
7. Perform a search in the URL bar.
8. Go to about:telemetry and note that there is no corresponding ”savant” event for the URL bar.
9. If you look at the previous main ping (not the current ping), you will see an ”end_study” ping with an object of ”expired”. The study expired during shutdown of the previous session.

Study expired between sessions

1. Go to about:config and create a new int pref with a name of “shield.savant.duration_override” and a value of 10000.
2. Set shield.savant.enabled to true.
3. Immediately close out of Firefox completely.
4. Wait 30 seconds.
5. Open Firefox.
6. Notice the ”end_study” event in about:telemetry with an object of ”expired”.

[biancadanforth]

Password Manager probes (Bug 1465685)

See Issue #20

Verify pwmgr - ask and pwmgr - save probes work:

1. In a new profile, launch Firefox and set the shield.savant.enabled pref to true.
2. Navigate to facebook.com
3. Enter in a made-up username and password (remember the username).
4. After you submit the login information, you’ll see a Password Manager panel prompt.
5. Click “Save”.
6. Go into about:telemetry#events-tab for the current ping. Observe there are two events, pwmgr - ask and pwmgr- save and both events have the same flow_id value in the extra field.

Verify pwmgr - ask and pwmgr - update probes work:

7. After performing steps 1-6 above, open Firefox in the same profile.
8. Navigate to facebook.com and enter the username from steps 3 above. Do not use the previously saved login information in the dropdown.
9. In the password field, type a completely new password and submit the login form.
10. Observe in about:telemetry#events-tab there are two events: pwmgr - ask and pwmgr - update and both events have the same flow_id value in the extra field.

Verify pwmgr_use - use works: Note: Unlike the other Password Manager probes, this event happens in the content as opposed to the parent process.

11. After performing steps 1 - 6 or 1 - 10 above, open Firefox in the same profile.
12. Navigate to facebook.com; your login information should be automatically filled in.
13. If not, as you enter the name, you will see a dropdown menu item appear with a small key icon. Select this menu item to auto-fill your login information.
14. Observe in about:telemetry#event-tab there is a new event: pwmgr_use - use.

Verify pwmgr - ask probe works:

16. Repeat steps 1-4 above.
17. Then click “Don’t Save”.
18. Go into about:telemetry#events-tab for the current ping, and click on “Events” in the side menu.
19. Observe there is only one event, pw_manager ask.

login_form probe (Bug 1465685)

See Issue #24

Verify login_form - load works:

1. Set shield.savant.enabled to true in about:config.
2. Visit any site with a login form (ex: amazon.com).
3. Observe one or more instances of the login_form event (see #24 for why there might be more than one event fired for a single login page) with a method of load in about:telemetry#events-tab for the most recent main ping.

Verify login_form - submit works:

1. Set shield.savant.enabled to true in about:config.
2. Visit any site with a login form (ex: amazon.com).
3. Enter phony (or actual) login information and submit the form.
4. Observe the login_form probe with a method of submit in about:telemetry#events-tab for the most recent main ping. Note the limitations of recording this event, as described in #24 , such as it will not fire for login forms that do not use a <form> element.

[biancadanforth]

Addon probes (Bug 1465707)

Note: Add-on install/uninstall events change the "environment", triggering the existing main ping with all of its appended events to fire, and a new main ping with no events appended to appear. So look for these events in both the current main ping and the previous main ping.

See Issue #25

Verify addon - install_start and addon - install_finish

1. Set shield.savant.enabled to true in about:config.
2. Go to addons.mozilla.org and install any add-on (Add-on A).
3. If the installation is successful, observe the addon probe with a method of install_start and install_finish in about:telemetry#events-tab. These two events should have the same add-on ID in the value field. Note this addon ID.

Verify value field addon ID is the same for events with the same add-on. && Verify addon - uninstall_start and addon - uninstall_finish

4. Repeat Steps 1-3 to install a different add-on (Add-on B) from addons.mozilla.org.
5. Observe in about:telemetry#events-tab that the addon - install_start and addon - install_finish event value in the value field for Add-on B are different than that for Add-on A.
6. Now uninstall Add-on A from step 2 through about:addons.
7. If the uninstall is successful, observe in about:telemetry#events-tab that the addon - uninstall_start and addon - uninstall_finish event value value matches that of the original value for Add-on A from Step 3 above. Note: the uninstall event will not fire until you close out of about:addons; this is a usability feature in case the user decides to undo the uninstall.

Verify addon - enable and addon - disable

8. Disable Add-on B from Step 4 via about:addons.
9. Enable Add-on B from Step 4 via about:addons.
10. Observe in about:telemetry#events-tab the addon probe with a method of enable and disable. These two events should have the same value in the value field.

[biancadanforth]

Bookmark probes (Bug 1465703)

See Issue #27

Verify bookmark - add

1. Set shield.savant.enabled to true in about:config.
2. Add a bookmark in any way you choose.
3. Observe in about:telemetry#events-tab the bookmark - save event has been captured.

Verify follow_bookmark - open

4. Open any bookmark in any way you choose.
5. Observe in about:telemetry#events-tab the follow_bookmark - open event has been captured.

Verify bookmark - remove

6. Remove the bookmark added from Step 2 (or any bookmark) in any way you choose.
7. Observe in about:telemetry#events-tab the bookmark - remove event has been captured.

[biancadanforth]

Tab probes (Bug 1465694)

See Issue #29

Verify tab - open

1. Set shield.savant.enabled to true in about:config.
2. Open a tab. Note that when a new tab is created, both the open and select events fire.
3. Observe in about:telemetry#events-tab the tab - open event has been captured.

Verify tab - close

4. Close a tab. Note: Detaching a tab and creating a new window with it also fires the close event.
5. Observe in about:telemetry#events-tab the tab - close event has been captured.

Verify tab - select

6. Switch to a different, open tab in the same window from your current tab. Note: Switching to a different tab in another window does not fire the select event.
7. Observe in about:telemetry#events-tab the tab - select event has been captured.

[biancadanforth]

Follow urlbar link probes (Bug 1465704)

See Issue #31 .

Verify follow_urlbar_link - bookmark

1. Set shield.savant.enabled to true in about:config.
2. Visit a website (e.g. mozilla.com) and add it as a bookmark. Note the site's url.
3. Open a new tab.
4. Close the previous tab with the bookmarked website from Step 2.
5. In the new tab's urlbar, start typing the bookmarked site's url from Step 2.
6. In the dropdown, select the bookmark result (has a star icon next to it).
7. Observe in about:telemetry#events-tab the follow_urlbar_link - bookmark event has been captured.

Verify follow_urlbar_link - history

1. Set shield.savant.enabled to true in about:config.
2. Visit a website (e.g. mozilla.com). Note the url for the site.
3. Open a new tab and close the previous tab from Step 1.
4. In the urlbar of the new tab, start typing the url. Select the history result from the dropdown (the result with the site's url that is NOT the first result in the dropdown).
5. Observe in about:telemetry#events-tab the follow_urlbar_link - history event has been captured.

[biancadanforth]

Reader Mode probes (Bug 1465698)

Note: These events take place in the content process; you'll need to switch the default of parent to content in the top right corner dropdown on about:telemetry#events-tab to view them.

See Issue #33 .

Verify readermode - on

1. Set shield.savant.enabled to true in about:config.
2. Visit a webpage that supports Reader Mode (e.g. a page with an <article> HTML element, like this medium.com article).
3. Turn on Reader Mode from the toolbar button in the urlbar.
4. Observe in about:telemetry#events-tab the readermode - on event has been captured in the content process.

Verify readermode - off

5. For the same webpage used in Step 2, turn off Reader Mode from the toolbar button in the urlbar.
6. Observe in about:telemetry#events-tab the readermode - off event has been captured in the content process.

[biancadanforth]

Menu probes (Bug 1465697)

See Issue #35 .

Verify library_menu - open

1. Set shield.savant.enabled to true in about:config.
2. Open the library menu through the library toolbar button (located to the left of the hamburger menu with the other browserAction buttons).
3. Observe in about:telemetry#events-tab the library_menu - open event has been captured.

Verify hamburger_menu - open

4. Open the hamburger menu.
5. Observe in about:telemetry#events-tab the hamburger_menu - open event has been captured.

Verify dotdotdot_menu - open

6. Open the dotdotdot menu (aka the pageAction menu) located at the far right in the urlbar.
7. Observe in about:telemetry#events-tab the dotdotdot_menu - open event has been captured.