Enhanced Tracking Protection needs Page Load Testing

mozilla-mobile / fenix

⚠️ Fenix (Firefox for Android) moved to a new repository. It is now developed and maintained as part of: https://github.com/mozilla-mobile/firefox-android

https://github.com/mozilla-mobile/firefox-android

Mozilla Public License 2.0

6.48k stars 1.27k forks source link

Enhanced Tracking Protection needs Page Load Testing #4783

Closed colintheshots closed 4 years ago

colintheshots commented 4 years ago

User Story

As a member of the Fenix team, I would like performance measurements on the relative page load speeds of Tracking Protection versus the new Enhanced Tracking Protection feature.

Dependencies

This will require some assistance from the Performance Team to learn how to run comparative Raptor testing using Try with live loads and custom parameters and/or APKs.

Acceptance Criteria

We have accurate page load performance measurements of ETP vs. TP on the latest Fenix with enough runs to have reasonable certainty in the results.

┆Issue is synchronized with this Jira Task

vesta0 commented 4 years ago

The goal of this experiment is to determine performance impact of switching from Strict TP (current) to Basic TP. Based on the results of this experiment, we will make a decision on whether Fenix default/recommended setting should be Basic or Strict (see here)

Cohort A:

Block Cross-site Tracking Cookies from Disconnect's Advertising, Analytics, and Social list

Cohort B (Current implementation):

Blocking tracking scripts (not just the tracking cookies) from Disconnect's Advertising, Analytics, and Social list

@Amejia481 and @ekager would you please add the name of lists/prefs to be used for each cohort? Thank you!

Amejia481 commented 4 years ago

Cohort a (ETP):

TrackingProtectionPolicy.strict()

Cohort B (Current TP implementation):

TrackingProtectionPolicy.recommended()

liuche commented 4 years ago

Remaining work E2.

vesta0 commented 4 years ago

Cohort a (ETP):

* `TrackingProtectionPolicy.strict()`

I don't believe this is correct. ETP (#1749) is a new category and refers to blocking Cross-site Tracking Cookies from Disconnect's Advertising, Analytics, and Social list. It is less strict than what we currently block in Fenix which is blocking tracking scripts/sub-resources, not just cookies.

Cohort B (Current TP implementation):
* `TrackingProtectionPolicy.recommended()`

Amejia481 commented 4 years ago

The default policies on ac have been updated on 12.0.0 to be aligned with the mockups and what Desktop is using https://github.com/mozilla-mobile/android-components/issues/4349.

Now the cohorts should match as follow:

Cohort A: (Standard on the mockups)

TrackingProtectionPolicy.recommended()

Cohort B: (Strict on the mockups)

TrackingProtectionPolicy.strict()

mcomella commented 4 years ago

Triage: waiting on any form of page load testing with visual metrics.

mcomella commented 4 years ago

From Eric on this issue:

Yeah, please close. Testing has happened already at least twice that I'm aware of.