brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.8k stars 2.33k forks source link

Add cookie consent notice blocking P3A metrics #25568

Closed DJAndries closed 2 years ago

DJAndries commented 2 years ago

Description

Add two P3A questions for cookie consent notice blocking settings.

"If you have viewed the cookie consent block prompt, how did you react?" Brave.Shields.CookieListPrompt

0) Have not seen the prompt 1) Have seen the prompt, but did not react 2) Clicked "No thanks" on the prompt 3) Clicked "Block cookie notices" on the prompt

"Do you have cookie consent notice blocking enabled?" Brave.Shields.CookieListEnabled

0) No 1) Yes

kjozwiak commented 2 years ago

The above requires 1.45.104 or higher for 1.45.x verification 👍

kjozwiak commented 2 years ago

Labelling as QA/Blocked as we'll need to wait for https://github.com/brave/brave-browser/issues/26078 to be fixed and uplifted into 1.45.x. We'll also need to enable BraveAdblockCookieListOptInStudy on BETA and Release before we can verify the above as we'll be enabling this feature via Grffin on both Desktop & Android.

@Uni-verse @srirambv I can run through this as I have the most context/been working on this for a while.

GeetaSarvadnya commented 2 years ago

Verification PASSED on

Brave | 1.45.109 Chromium: 107.0.5304.62 (Official Build) (64-bit)
-- | --
Revision | 1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}
OS | Windows 10 Version 21H2 (Build 19044.2130)

Brave.Shields.CookieListPrompt

Brave.Shields.CookieListPrompt = 0_PASSED 1. Clean profile 1.46.x 2. Open brave://local-state file 3. Ensured that the metrics `Brave.Shields.CookieListPrompt` is NOT shown
Brave.Shields.CookieListPrompt = 1_PASSED
Case 1: Click on close button on cookie consent block prompt 1. Clean profile 1.45.109 4. Enable the flag "Show an opt-in bubble for the "Easylist-Cookie List" filter" via brave://flags 5. Relaunch the browser 6. Ensure that the `cookie consent block` prompt is shown 7. Click on the `x` or close button in the popup 8. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![image](https://user-images.githubusercontent.com/38657976/197253597-4f3b90f6-dc95-4a4d-ade7-a587a0880297.png)
Case 2: Open a new NTP when cookies consent modal popup prompt 1. Clean profile 1.45.109 2. Enable the flag "Show an opt-in bubble for the "Easylist-Cookie List" filter" via brave://flags 4. Relaunch the browser 5. Ensure that the `cookie consent block` prompt is shown 6. Open a new NTP and ensure cookies consent popup is dismissed 7. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![image](https://user-images.githubusercontent.com/38657976/197253653-68de500c-6b53-4795-913b-1a1043474ed6.png)
Case 3: Close the `cookie consent block` modal popup tab 1. Clean profile 1.45.109 2. Open any site in NTP 3. Enable the flag "Show an opt-in bubble for the "Easylist-Cookie List" filter" via brave://flags 4. Relaunch the browser 5. Ensure that the `cookie consent block` prompt is shown 6. Close the active tab of `cookie consent block` prompt 9. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![image](https://user-images.githubusercontent.com/38657976/197253879-cb129c0d-8157-458c-abf1-6f53a5ad03b3.png)
Case 4: Switch the tab to other tab when cookies consent modal popup prompt 1. Clean profile 1.45.109 2. Open any site in NTP 3. Enable the flag "Show an opt-in bubble for the "Easylist-Cookie List" filter" via brave://flags 4. Relaunch the browser 5. Ensure that the `cookie consent block` prompt is shown 6. Switch the tab to a site ( opened in step 2) 8. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![image](https://user-images.githubusercontent.com/38657976/197254414-a0a23ad2-7f55-410d-b69b-cfc441afee1d.png)
Brave.Shields.CookieListPrompt = 2_PASSED - Confirmed that the response value for the metrics `Brave.Shields.CookieListPrompt` is 2 when user clicks on the "No thanks" button on the `cookie consent block` prompt ![image](https://user-images.githubusercontent.com/38657976/197256184-f8a86018-5f32-48b1-9b77-ed803225775e.png)
Brave.Shields.CookieListPrompt = 3_PASSED - Confirmed that the response value for the metrics `Brave.Shields.CookieListPrompt` is 3 when user clicks on the "Block cookie notices" button on the `cookie consent block` prompt ![image](https://user-images.githubusercontent.com/38657976/197256515-211b22c4-1dd0-4296-831b-7a6012b706ea.png)

Brave.Shields.CookieListEnabled

Brave.Shields.CookieListEnabled = 0_PASSED
Case 1: Have not enabled the flag "Easylist-Cookie List"_PASSED 1. Clean profile 1.45.109 2. Open brave://local-state file 3. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` ![image](https://user-images.githubusercontent.com/38657976/197257478-ad664f8f-6fc9-4fd4-a16e-5dac76265f70.png)
Case 2: Enabled the flag "Easylist-Cookie List" and clicked on Close button_PASSED 1. Clean profile 1.45.109 2. Open brave://flags and enable the flag `Show an opt-in bubble for the "Easylist-Cookie List" filter` 3. Relaunch the browser 4. Confirmed that the cookies consent prompt is shown 5. Click on close (x) button 6. Open brave://local-state file 8. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` and `Brave.Shields.CookieListPrompt` response value is `1` ![image](https://user-images.githubusercontent.com/38657976/197258425-9cd0eb5e-0456-4422-9f16-b0b8d54b2bc7.png)
Case 3: Enabled the flag "Easylist-Cookie List" and closed the cookies consent tab_PASSED 1. Clean profile 1.45.109 2. Open brave://local-state 3. Open brave://flags and enable the flag `Show an opt-in bubble for the "Easylist-Cookie List" filter` 4. Relaunch the browser 5. Confirmed that the cookies consent prompt is shown 6. Close the cookies consent tab 7. Reload brave://local-state file 10. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` and `Brave.Shields.CookieListPrompt` response value is `1` ![image](https://user-images.githubusercontent.com/38657976/197258746-54d5f327-4764-42d3-bedb-7d1276b3d882.png)
Case 4: Enabled the flag "Easylist-Cookie List" and switch to a tab other than the cookies consent tab_PASSED 1. Clean profile 1.45.109 2. Open brave://local-state 3. Open brave://flags and enable the flag `Show an opt-in bubble for the "Easylist-Cookie List" filter` 4. Relaunch the browser 5. Confirmed that the cookies consent prompt is shown 6. Switch to a tab other than the cookies consent tab 7. Reload brave://local-state file 9. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` and `Brave.Shields.CookieListPrompt` response value is `1` ![image](https://user-images.githubusercontent.com/38657976/197259008-e2365296-0912-4ba8-8ca0-c2a4dbc9bb7f.png)
Brave.Shields.CookieListEnabled = 1_PASSED 1. Clean profile 1.45.109 2. Open brave://flags and enable the flag `Show an opt-in bubble for the "Easylist-Cookie List" filter` 3. Relaunch the browser 4. Confirmed that the cookies consent prompt is shown 5. Click on the "Block cookie notices" on the prompt 7. Reload brave://local-state file 8. Confirmed that the metrics `Brave.Shields.CookieListEnabled` response value is `1` when user clicks on the "Block cookie notices"on the prompt and the metrics `Brave.Shields.CookieListPrompt` response value is `3` 11. Confirmed that the `Easylist - Cookie List` is enabled in brave://settings/shields/filters when user clicks on the "Block cookie notices" on the prompt Example | Example -- | -- ![image](https://user-images.githubusercontent.com/38657976/197259356-f57caf3d-536a-4c70-a1b9-8e3eeae9ac40.png) | ![image](https://user-images.githubusercontent.com/38657976/197259494-a4490484-b253-4c1a-a188-e67bdd39f4f0.png)
kjozwiak commented 2 years ago

Verification PASSED on Samsung S10+ running Android 12 using the following build(s):

Brave | 1.45.111 Chromium: 107.0.5304.62 (Official Build) (32-bit)
--- | ---
Revision | 1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}
OS | Android 12; Build/SP1A.210812.016

Brave.Shields.CookieListPrompt

Brave.Shields.CookieListPrompt = 0_PASSED 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open `brave://local-state` 3. Ensured that the `Brave.Shields.CookieListPrompt` metric is NOT being shown/displayed ![Screenshot_20221024-102527_Brave](https://user-images.githubusercontent.com/2602313/197551731-5d09e921-d9b9-445a-bd9e-138a78ad1536.jpg)
Brave.Shields.CookieListPrompt = 1_PASSED
Case 1: Click on close button on cookie consent block prompt This isn't a valid case on `Android` as there's no `X` via the `cookie consent` panel/notification.
Case 2: Open a new NTP when cookies consent modal popup prompt 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 5. Relaunch the browser 6. Ensure that the `cookie consent block` prompt is shown after 5 refresh/loads 7. Open a new NTP and ensure cookies consent popup is dismissed 8. Open `brave://local-state` file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![Screenshot_20221024-102803_Brave](https://user-images.githubusercontent.com/2602313/197551665-65414d20-b445-482a-b84e-d0f81d772bc7.jpg)
Case 3: Close the `cookie consent block` modal popup tab This isn't a valid case on `mobile/phone` as you can't close a tab unless you go into the `Tab View` which closes the panel/notification. Ran through this particular case on a `Samsung Tab S8 Ultra`. 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open any site in NTP 3. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 4. Relaunch the browser 5. Ensure that the `cookie consent block` prompt is shown 6. Close the active tab of `cookie consent block` prompt 9. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![Screenshot_20221024_115454_Brave](https://user-images.githubusercontent.com/2602313/197571226-9c05659e-34ff-46df-b2a9-252ab7cd9def.jpg)
Case 4: Switch the tab to other tab when cookies consent modal popup prompt 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open any site in NTP 3. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 4. Relaunch the browser 5. Ensure that the `cookie consent block` prompt is shown after 5 refresh/loads 6. Switch the tab to a site ( opened in step 2) 9. Open brave://local-state file and ensured that the metrics response value for the metrics `Brave.Shields.CookieListPrompt` is `1` ![Screenshot_20221024-103648_Brave](https://user-images.githubusercontent.com/2602313/197553354-fb5ff84d-e2ed-423e-8c1c-77a72b59e9c4.jpg)
Brave.Shields.CookieListPrompt = 2_PASSED Confirmed that the response value for the metrics `Brave.Shields.CookieListPrompt` is 2 when user clicks on the "No thanks" button on the `cookie consent block` prompt as per the following: ![Screenshot_20221024-104357_Brave](https://user-images.githubusercontent.com/2602313/197554614-77aece58-b0e6-4a50-985e-99a4581c079c.jpg)
Brave.Shields.CookieListPrompt = 3_PASSED Confirmed that the response value for the metrics `Brave.Shields.CookieListPrompt` is 3 when user clicks on the "Block cookie notices" button on the `cookie consent block` prompt as per the following: ![Screenshot_20221024-104619_Brave](https://user-images.githubusercontent.com/2602313/197555219-d39ffed6-f5e1-45c1-83be-33da6058602e.jpg)

Brave.Shields.CookieListEnabled

Brave.Shields.CookieListEnabled = 0_PASSED
Case 1: Have not enabled the flag "Easylist-Cookie List"_PASSED 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open `brave://local-state` 3. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` ![Screenshot_20221024-104951_Brave](https://user-images.githubusercontent.com/2602313/197556061-170425b0-93d6-47f6-84f2-b6bb15b8cb7f.jpg)
Case 2: Enabled the flag "Easylist-Cookie List" and clicked on Close button_PASSED This isn't a valid case on `Android` as there's no `X` via the `cookie consent` panel/notification.
Case 3: Enabled the flag "Easylist-Cookie List" and closed the cookies consent tab_PASSED 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open `brave://local-state` 3. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 4. Relaunch the browser 5. Confirmed that the cookies consent prompt is shown 6. Close the cookies consent tab 7. Reload the `brave://local-state` tab 11. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` and `Brave.Shields.CookieListPrompt` response value is `1` ![Screenshot_20221024-105504_Brave](https://user-images.githubusercontent.com/2602313/197558243-35b02905-2db8-4552-9dec-e26313cd0c2e.jpg)
Case 4: Enabled the flag "Easylist-Cookie List" and switch to a tab other than the cookies consent tab_PASSED 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Open `brave://local-state` 3. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 4. Relaunch the browser 5. Confirmed that the cookies consent prompt is shown 6. Switch to a tab other than the cookies consent tab 7. Reload the `brave://local-state` tab 10. Ensured that the metrics `Brave.Shields.CookieListEnabled` response value is `0` and `Brave.Shields.CookieListPrompt` response value is `1` ![Screenshot_20221024-105504_Brave](https://user-images.githubusercontent.com/2602313/197557469-b695d554-d6ec-4ec7-9bab-04819203beb2.jpg)
Brave.Shields.CookieListEnabled = 1_PASSED 1. Clean `1.45.111 Chromium: 107.0.5304.62` profile 2. Enable `brave://flags#brave-adblock-cookie-list-opt-in` via `brave://flags` 3. Relaunch the browser 4. Confirmed that the cookies consent prompt is shown 5. Click on the "Block cookie notices" on the prompt 7. Reload brave://local-state file 8. Confirmed that the metrics `Brave.Shields.CookieListEnabled` response value is `1` when user clicks on the "Block cookie notices"on the prompt and the metrics `Brave.Shields.CookieListPrompt` response value is `3` 12. Confirmed that the `Easylist - Cookie List` is enabled in brave://settings/shields/filters when user clicks on the "Block cookie notices" on the prompt **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- ![Screenshot_20221024-110127_Brave](https://user-images.githubusercontent.com/2602313/197559484-ee172238-0feb-41c9-a4bd-eb72078a1072.jpg) | ![Screenshot_20221024-110138_Brave](https://user-images.githubusercontent.com/2602313/197559399-510cda26-7938-42e8-9edf-4a12f4dd2b4c.jpg) | ![Screenshot_20221024-110151_Brave](https://user-images.githubusercontent.com/2602313/197559534-6501b999-b4ba-44d8-9755-c8b020941ab0.jpg)