search

brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.57k stars 2.28k forks source link

Simplify variations pull requests testing process #39689

Closed goodov closed 2 months ago

goodov commented 2 months ago

Description

Add support for --variations-pr=<pull_number> command line flag that will automatically set required flags to fetch the test seed generated as a part of https://github.com/brave/brave-variations pull request.

Related https://github.com/brave/brave-browser/issues/33654

goodov commented 2 months ago

Test cases

Only one desktop platform can be tested here (mac/linux/windows), don't need to test all of them. Android and iOS should be tested both.

By default https://variations.brave.com/seed is used

  1. Launch browser
  2. Open brave://version
  3. Expect 
    --variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed

    is present in "Command Line" field.

  4. Expect production variations are listed in "Active Variations" field.

Setting a test seed via --variations-pr command line flag

For the test we will use a test seed from this draft PR: https://github.com/brave/brave-variations/pull/1123 The seed contains a single study BraveAdsAdEventStudy enabled on all platforms.

  1. Launch browser with --variations-pr=1123 parameter (on Android use debug menu to set command line; on iOS set "Variations PR" parameter in Brave-core debug menu).
  2. Restart browser
  3. Wait 10 seconds
  4. Restart browser
  5. Open brave://version
  6. Expect 
    --accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed

    is present in "Command Line" field.

  7. Expect "Seed Details > Version" from this comment is present in "Active Variations" field.

Ensure https://variations.brave.com/seed is restored

  1. Remove the parameter from the command line/debug menu.
  2. Restart browser
  3. Wait 10 second
  4. Restart browser
  5. Expect 
    --variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed

    is present in "Command Line" field.

  6. Expect these lines are NOT present in "Command Line" field: 
    --accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
  7. Expect production variations are listed in "Active Variations" field.
kjozwiak commented 2 months ago

The above requires 1.68.124 or higher for 1.68.x verification 👍 @brave/qa-team please use https://github.com/brave/brave-core/pull/24604#issuecomment-2229417488 or https://github.com/brave/brave-core/pull/24604#issuecomment-2231943968 as the template. Would recommend using https://github.com/brave/brave-core/pull/24604#issuecomment-2231943968 as it has several other cases for Desktop like an upgrade case that should also be checked.

stephendonner commented 2 months ago

Verification PASSED using

Brave | 1.68.124 Chromium: 127.0.6533.57 (Official Build) (x86_64)
-- | --
Revision | d048917c488542dc56e67059011b2cf9d6043bb9
OS | macOS Version 11.7.10 (Build 20G1427)

Test Case #1 - (universal executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` via the `standalone` executable and visited `brave://version` * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- Screen Shot 2024-07-18 at 11 05 29 AM | Screen Shot 2024-07-18 at 11 06 30 AM | Screen Shot 2024-07-18 at 11 06 47 AM * restarted `1.68.124 Chromium: 127.0.6533.57` and ensured that the studies/variations were being pulled/applied * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- Screen Shot 2024-07-18 at 11 08 12 AM | Screen Shot 2024-07-18 at 11 08 29 AM | Screen Shot 2024-07-18 at 11 08 42 AM

Test Case #2 - (x64 executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` via the `x64` online executable and visited `brave://version` * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that the studies/variations were being pulled/applied via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- Screen Shot 2024-07-18 at 11 16 42 AM | Screen Shot 2024-07-18 at 11 17 06 AM | Screen Shot 2024-07-18 at 11 17 13 AM

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` using `--enable-logging=stderr --variations-pr=1123` * ensured that there's no variations being displayed via `brave://version` (requires a restart to apply the pulled variation) * re-launched using `--enable-logging=stderr --variations-pr=1123` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` **`Example`** | **`Example`** ---------------|---------------- Screen Shot 2024-07-18 at 11 20 52 AM | Screen Shot 2024-07-18 at 11 21 39 AM

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` and started `1.68.124 Chromium: 127.0.6533.57` using `--enable-logging=stderr`. * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- Screen Shot 2024-07-18 at 11 25 32 AM | Screen Shot 2024-07-18 at 11 25 45 AM

Test Case #5 - Upgrade from 1.67.134 Chromium: 126.0.6478.186 -> 1.68.124 Chromium: 127.0.6533.57 - PASSED

STR/Cases: * installed `1.67.134 Chromium: 126.0.6478.186` and relaunched so the studies are applied/visible via `brave://version` * upgraded to `1.68.124 Chromium: 127.0.6533.57` via `brave://settings/help` by launching with `./Brave\ Browser --update-feed-url=https://updates.bravesoftware.com/sparkle/Brave-Browser/test/appcast.xml` * ensured that once updated, the variations/studies are still visible/being applied via `brave://version` `1.67.134 Chromium: 126.0.6478.186` | `update` | `1.68.124 Chromium: 127.0.6533.57` ---------|----------|-------- Screen Shot 2024-07-18 at 11 58 35 AM | Screen Shot 2024-07-18 at 11 59 23 AM | Screen Shot 2024-07-18 at 12 00 18 PM
btlechowski commented 2 months ago

Verified with

Brave 1.68.124 Chromium: 127.0.6533.57 (Official Build) (64-bit)
Revision d048917c488542dc56e67059011b2cf9d6043bb9
OS Linux

Test Case #1 - (offline executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * launched `1.69.x` via the `standalone` executable and visited `brave://version` * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` ![image](https://github.com/user-attachments/assets/0f052b8d-700e-4d84-b954-31f075c85ab7) * restarted `1.69.x` and ensured that the studies/variations were being pulled/applied * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` ![image](https://github.com/user-attachments/assets/0d46075b-1255-4093-b868-3c40b9840ec9)

Test Case #2 - (online using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that the studies/variations were being pulled/applied via `brave://version` ![image](https://github.com/user-attachments/assets/e68049db-1f36-4356-a72f-78f19b0cc4cc)

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases: * launched `1.69.x` using `--enable-logging=stderr --variations-pr=1123` * ensured that there's no variations being displayed via `brave://version` (requires a restart to apply the pulled variation) * re-launched using `brave.exe --enable-logging=stderr --variations-pr=1123` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` ![image](https://github.com/user-attachments/assets/de00bc9d-0dec-4438-8cd3-394246e69b07)

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` and started `1.69.111 Chromium: 127.0.6533.43` using `brave.exe --enable-logging=stderr`. * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` ![image](https://github.com/user-attachments/assets/74b2b6e1-51a4-4ae6-af9c-ff8e32dbf033)

Test Case #5 - Upgrade from 1.69.102 Chromium: 127.0.6533.43 -> 1.69.110 Chromium: 127.0.6533.43 - PASSED

STR/Cases: * installed `1.69.102 Chromium: 127.0.6533.43` and relaunched so the studies are applied/visible via `brave://version` * upgraded to `1.69.110 Chromium: 127.0.6533.43` via `brave://settings/help` * ensured that once updated, the variations/studies are still visible/being applied via `brave://version` ![image](https://github.com/user-attachments/assets/4c2514fb-f97e-40eb-93ac-84992c9967d1)
MadhaviSeelam commented 2 months ago

Verification Passed using

Brave | 1.68.124 Chromium: 127.0.6533.57 (Official Build) (64-bit)
-- | --
Revision | d048917c488542dc56e67059011b2cf9d6043bb9
OS | Windows 11 Version 23H2 (Build 22631.3880)

Test Case #1 - (offline executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` via the `standalone` executable and visited `brave://version` * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- ![image](https://github.com/user-attachments/assets/91ea0c43-4a14-4d27-bebe-dd1c604d3dd9)|image|image * restarted `1.68.124 Chromium: 127.0.6533.57` and ensured that the studies/variations were being pulled/applied * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- image|![image](https://github.com/user-attachments/assets/93ae1e66-235f-4289-8682-b790728ae84f)|![image](https://github.com/user-attachments/assets/6b53cc5f-ed5c-4b6b-96d1-61669578e898)

Test Case #2 - (online stub executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` via the `stub` online executable and visited `brave://version` * ensured that `--variations-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that `--variations-insecure-server-url=https://variations.brave.com/seed` is visible via `brave://version` * ensured that the studies/variations were being pulled/applied via `brave://version` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- ![image](https://github.com/user-attachments/assets/e49a1902-41f7-4dd4-9d2d-17feca992068)|![image](https://github.com/user-attachments/assets/a0c81a8f-be2d-4e8f-857e-7e5542b03492)|![image](https://github.com/user-attachments/assets/ca66eb70-6485-4b02-aaca-a1821cf92c19)

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases: * launched `1.68.124 Chromium: 127.0.6533.57` using `brave.exe --enable-logging=stderr --variations-pr=1123` * ensured that there's no variations being displayed via `brave://version` (requires a restart to apply the pulled variation) * re-launched using `brave.exe --enable-logging=stderr --variations-pr=1123` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` **`Example`** | **`Example`** ---------------|---------------- ![image](https://github.com/user-attachments/assets/d892b5b1-27f2-4235-ab76-46cdca5fa1e9)|![image](https://github.com/user-attachments/assets/eb6d45ef-e008-43b7-b9f4-a6d460ff78d9)

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` and started `1.68.124 Chromium: 127.0.6533.57` using `brave.exe --enable-logging=stderr`. * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- ![image](https://github.com/user-attachments/assets/536b87aa-a26f-41a6-a988-1cff585fccc2)|![image](https://github.com/user-attachments/assets/28d588cc-321a-484c-98c9-3a0484771fe4)

Test Case #5 - Upgrade from 1.67.134 Chromium: 126.0.6478.186 -> 1.68.124 Chromium: 127.0.6533.57 - PASSED

STR/Cases: * installed `1.67.134 Chromium: 126.0.6478.186` and relaunched so the studies are applied/visible via `brave://version` * upgraded to `1.68.124 Chromium: 127.0.6533.57` via `brave://settings/help` * ensured that once updated, the variations/studies are still visible/being applied via `brave://version` `1.67.134 Chromium: 126.0.6478.186`| `1.68.124 Chromium: 127.0.6533.57` ----------------------------------------|------------------------------------------ ![image](https://github.com/user-attachments/assets/3bf1d3e6-bda3-458b-9290-d825a48859fa)|![image](https://github.com/user-attachments/assets/78cc428c-481a-4482-a2c1-16808d1b987a)
kjozwiak commented 1 month ago

Verification PASSED on Pixel 6 Pro running Android 15 using the following build(s):

Brave | 1.68.125 Chromium: 127.0.6533.57 (Official Build) (64-bit)
--- | ---
Revision | 14ff4d467fb3eec4e6fc0a2f3e2d3f7b4c8cb788
OS | Android 15; Build/AP31.240617.009; 35; REL

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases: * installed/launched `1.68.125 Chromium: 127.0.6533.57` * visited `brave://version` and ensured that the following parameters/switches were being used via `Command Line` * ensured that `production` variations/studies were being displayed via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|--------------- ![Screenshot_20240723-014318](https://github.com/user-attachments/assets/9ddc4277-fe7d-419a-b45d-d3ce9b94f886) | ![Screenshot_20240723-014313](https://github.com/user-attachments/assets/c247e784-9ccd-4544-bad5-7ee4ea814541) | ![Screenshot_20240723-014256](https://github.com/user-attachments/assets/cc7727fc-db53-4d95-b1cf-2e2340555d51)

Test Case #2 - Setting test seed using --variations-pr= - PASSED

**`Prerequisite:`** ensured that the network on the device has been disabled before setting `--variations-pr=` STR/Cases: * installed/launched `1.68.125 Chromium: 127.0.6533.57` * added `--variations-pr=`1123` via `Command Line String` under the `QA Developer` settings * closed Brave and re-enabled the devices network/WiFi * visited `brave://version` and ensured there was nothing being listed under `Variations` via `brave://version` * restarted `1.68.125 Chromium: 127.0.6533.57` once again after waiting `~10s` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------|------------------|-------------- ![Screenshot_20240723-014749](https://github.com/user-attachments/assets/18a5f630-ab94-4b33-886d-9e8457cbda5b) | ![Screenshot_20240723-014823](https://github.com/user-attachments/assets/20a54b52-ee27-49d9-b094-e4557feeaace) | ![Screenshot_20240723-014908](https://github.com/user-attachments/assets/91f3f3d6-113b-434d-a478-de05c4d20d74) | ![Screenshot_20240723-014924](https://github.com/user-attachments/assets/29f739c5-10e8-41d1-8cbb-3fd2a3a509ae) | ![Screenshot_20240723-014941](https://github.com/user-attachments/assets/d1f78de2-84c7-4421-aff5-83ced21e128f) | ![Screenshot_20240723-014959](https://github.com/user-attachments/assets/a74ab8d9-ecc6-4f87-a5ff-47967d16f7a7) * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` ![Screenshot_20240723-015009](https://github.com/user-attachments/assets/32bf7983-b837-4820-bc9b-a9876b5001da)

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` from the `Command Line String` under the `QA Developer` settings and restarted `1.69.113 Chromium: 127.0.6533.43` * waited `~10s` and restarted `1.69.113 Chromium: 127.0.6533.43` once again * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------- ![Screenshot_20240723-015328](https://github.com/user-attachments/assets/6d6798fe-c444-486a-9205-58bcd8471173) | ![Screenshot_20240723-015443](https://github.com/user-attachments/assets/228ad013-eb73-4af6-b5b6-a133358b2acb) | ![Screenshot_20240723-015512](https://github.com/user-attachments/assets/efb36775-155d-41bb-a3e3-ad937746de28) | ![Screenshot_20240723-015531](https://github.com/user-attachments/assets/86bd8eab-516b-4eee-a7be-ce9f33dff148) * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- ![Screenshot_20240723-015645](https://github.com/user-attachments/assets/de5eac09-af0c-4755-b255-59b2135966f8) | ![Screenshot_20240723-015633](https://github.com/user-attachments/assets/3cf342fd-97d3-412b-a862-cbbd5a5c7eb9)

Verification PASSED on Samsung Galaxy Tab S8 Ultra running Android 14 using the following build(s):

Brave | 1.68.125 Chromium: 127.0.6533.57 (Official Build) (64-bit)
--- | ---
Revision | 14ff4d467fb3eec4e6fc0a2f3e2d3f7b4c8cb788
OS | Android 14; Build/UP1A.231005.007; 34; REL

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases: * installed/launched `1.68.125 Chromium: 127.0.6533.57` * visited `brave://version` and ensured that the following parameters/switches were being used via `Command Line` * ensured that `production` variations/studies were being displayed via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|--------------- ![Screenshot_20240723_020438_Brave](https://github.com/user-attachments/assets/feb63fc3-b2c5-43d6-9af5-c68cd50b3964) | ![Screenshot_20240723_020531_Brave](https://github.com/user-attachments/assets/da69178b-279e-4958-b393-48609638881d) | ![Screenshot_20240723_020548_Brave](https://github.com/user-attachments/assets/3e7f4b3b-f1f8-4080-b0af-5143853c2a34)

Test Case #2 - Setting test seed using --variations-pr= - PASSED

**`Prerequisite:`** ensured that the network on the device has been disabled before setting `--variations-pr=` STR/Cases: * installed/launched `1.68.125 Chromium: 127.0.6533.57` * added `--variations-pr=`1123` via `Command Line String` under the `QA Developer` settings * closed Brave and re-enabled the devices network/WiFi * visited `brave://version` and ensured there was nothing being listed under `Variations` via `brave://version` * restarted `1.68.125 Chromium: 127.0.6533.57` once again after waiting `~10s` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------|------------------|-------------- ![Screenshot_20240723_020929_Brave](https://github.com/user-attachments/assets/5c4cd108-cd77-41da-b0b3-61509ac6fcb4) | ![Screenshot_20240723_021012_Brave](https://github.com/user-attachments/assets/dffb149d-a757-48ae-81a5-79b84d221fbf) | ![Screenshot_20240723_021113_Brave](https://github.com/user-attachments/assets/47b6ecd0-e3bb-4b05-8150-3033882a6f73) | ![Screenshot_20240723_021125_Brave](https://github.com/user-attachments/assets/8cbb8e60-3f88-498d-995e-228e465ddc97) | ![Screenshot_20240723_021141_Brave](https://github.com/user-attachments/assets/e80e25d3-4ce3-42af-a8d6-5cb3d29a477a) | ![Screenshot_20240723_021213_Brave](https://github.com/user-attachments/assets/87bb3875-d6d3-494b-bab4-5f3c53052ce4) * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` ![Screenshot_20240723_021223_Brave](https://github.com/user-attachments/assets/f0ebbdf6-8217-450e-8506-95b49a026e63)

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` from the `Command Line String` under the `QA Developer` settings and restarted `1.69.113 Chromium: 127.0.6533.43` * waited `~10s` and restarted `1.69.113 Chromium: 127.0.6533.43` once again * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------- ![Screenshot_20240723_021505_Brave](https://github.com/user-attachments/assets/16616693-12ad-49ee-9aca-daba11d16603) | ![Screenshot_20240723_021551_Brave](https://github.com/user-attachments/assets/80a706df-6c04-4ce3-818e-290673742cb3) | ![Screenshot_20240723_021608_Brave](https://github.com/user-attachments/assets/bc906a72-e5f2-4627-a972-342623976514) | ![Screenshot_20240723_021626_Brave](https://github.com/user-attachments/assets/4973d13c-fe15-45fe-82ca-ada84b55870a) * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- ![Screenshot_20240723_021658_Brave](https://github.com/user-attachments/assets/e3027d54-c2ed-409c-9ef7-15840e0989a6) | ![Screenshot_20240723_021713_Brave](https://github.com/user-attachments/assets/76a292ac-6d07-4c8d-a678-5f0d5b7a8f05)
kjozwiak commented 1 month ago

Verification PASSED on iPhone 11 running iOS 17.5.1 using the following build(s):

Brave | 1.68.130 Chromium: 127.0.6533.73 (Official Build) stable (64-bit) 
--- | ---
Revision | 841d137ffede...
OS | iOS

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases: * installed `1.68.130 Chromium: 127.0.6533.73` via TF * visited `brave://version` and ensured that there's no studies/variations being listed * restarted `1.68.130 Chromium: 127.0.6533.73` once again * visited `brave://version` and ensured that the following parameters/switches were being used via `Command Line` * ensured that `production` variations/studies were being displayed via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|---------------- ![IMG_0417](https://github.com/user-attachments/assets/8e191915-a6b6-47fe-9d43-6d059eea6128) | ![IMG_0419](https://github.com/user-attachments/assets/48d0787b-1ba3-4ac6-b230-ee1919675580) | ![IMG_0418](https://github.com/user-attachments/assets/324d41b5-0592-424e-afae-12101b7f3b03)

Test Case #2 - Setting test seed using --variations-pr= - PASSED (using provided switch)

**`Prerequisite:`** ensured that the network on the device has been disabled before setting `--variations-pr=` STR/Cases: * installed `1.68.130 Chromium: 127.0.6533.73` via TF * enabled the `Variations PR` switch under `BraveCore Switches` and entered `1123` as the value * closed Brave and re-enabled the devices network/WiFi * visited `brave://version` and ensured there was nothing being listed under `Variations` via `brave://version` * restarted `1.68.130 Chromium: 127.0.6533.73` once again after waiting `~10s` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------|-----------------|---------------- ![IMG_0420](https://github.com/user-attachments/assets/350a09db-2f59-4e3c-a211-f3a00526344b) | ![IMG_0421](https://github.com/user-attachments/assets/d04e862d-c3cb-4188-b2ea-d851032cfc7c) | ![IMG_0422](https://github.com/user-attachments/assets/4d305981-f9f6-474a-ae70-30a45212be6f) | ![IMG_0423](https://github.com/user-attachments/assets/93d07268-91d7-4e3d-a7f9-f44272c5ecfa) | ![IMG_0424](https://github.com/user-attachments/assets/7e56b9a6-e39b-4ad0-8b6b-c1c886370685) | ![IMG_0425](https://github.com/user-attachments/assets/59c9f801-57a8-4421-afab-6f5ae1b8eef2) * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` ![IMG_0426](https://github.com/user-attachments/assets/c8ab8d31-1ddd-4910-a9bf-0351c1b7501a)

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` from the `Command Line String` under the `QA Developer` settings and restarted `1.68.130 Chromium: 127.0.6533.73` * waited `~10s` and restarted `1.68.130 Chromium: 127.0.6533.73` once again * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|----------------|------------------ ![IMG_0427](https://github.com/user-attachments/assets/02fe3713-391d-4e5a-8d3f-81852aa484d5) | ![IMG_0428](https://github.com/user-attachments/assets/606fb657-1e0f-4c3c-9846-60a33c55651e) | ![IMG_0429](https://github.com/user-attachments/assets/03999901-4e53-48b0-9127-9cdbc8a45cd2) | ![IMG_0430](https://github.com/user-attachments/assets/fa0340ed-19df-484f-8f15-7dc43e7f6aa1) | ![IMG_0431](https://github.com/user-attachments/assets/bf117e6d-21fb-45f6-8574-a9a3c422c103) * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- ![IMG_0432](https://github.com/user-attachments/assets/2103b667-b981-486c-b143-dee1d1935e0d) | ![IMG_0433](https://github.com/user-attachments/assets/061cbd8d-d918-488b-8471-009d8bd5fcd6)

Verification PASSED on iPad Air (3rd Gen) running iOS 17.6 using the following build(s):

Brave | 1.68.130 Chromium: 127.0.6533.73 (Official Build) stable (64-bit) 
--- | ---
Revision | 841d137ffede0c1166265873663e6ebc56f93aa6
OS | iOS

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases: * installed `1.68.130 Chromium: 127.0.6533.73` via TF * visited `brave://version` and ensured that there's no studies/variations being listed * restarted `1.68.130 Chromium: 127.0.6533.73` once again * visited `brave://version` and ensured that the following parameters/switches were being used via `Command Line` * ensured that `production` variations/studies were being displayed via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|---------------- ![IMG_0412](https://github.com/user-attachments/assets/5faab5dd-d62f-4d5e-903d-8f33b4e70df4) | ![IMG_0413](https://github.com/user-attachments/assets/c7719ab2-daf9-4516-a1f8-3d2c91b0f2ba) | ![IMG_0414](https://github.com/user-attachments/assets/48992dec-93dd-494f-a315-4a5b0136313c)

Test Case #2 - Setting test seed using --variations-pr= - PASSED (using provided switch)

**`Prerequisite:`** ensured that the network on the device has been disabled before setting `--variations-pr=` STR/Cases: * installed `1.68.130 Chromium: 127.0.6533.73` via TF * enabled the `Variations PR` switch under `BraveCore Switches` and entered `1123` as the value * closed Brave and re-enabled the devices network/WiFi * visited `brave://version` and ensured there was nothing being listed under `Variations` via `brave://version` * restarted `1.68.130 Chromium: 127.0.6533.73` once again after waiting `~10s` * ensured that the following parameters/switches are present/visible under `Command Line` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling --variations-server-url=https://griffin.brave.com/pull/1123/seed --variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|---------------|-----------------|---------------- ![IMG_0417](https://github.com/user-attachments/assets/b333928e-e243-431f-8680-226ca366cfb0) | ![IMG_0418](https://github.com/user-attachments/assets/44cceb46-b627-4365-a297-3d6873d5b3ab) | ![IMG_0419](https://github.com/user-attachments/assets/8f566ca3-a2a3-4a6f-a75b-06092f20650c) | ![IMG_0415](https://github.com/user-attachments/assets/1e29283a-de1a-42a3-bdf2-0cf2232534e3) | ![IMG_0416](https://github.com/user-attachments/assets/7809de14-a471-4501-9ed2-625b29ed35b7) | ![IMG_0420](https://github.com/user-attachments/assets/5154251f-3b2c-46cb-bb1b-b799517a6a95) * ensured that `pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled` is being displayed under `Active variations:` via `brave://version` ![IMG_0421](https://github.com/user-attachments/assets/545c8f05-ec64-4471-a4e5-c0b62a5c416f)

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases: * after running through `Test Case #2` above, removed `--variations-pr=1123` from the `Command Line String` under the `QA Developer` settings and restarted `1.68.130 Chromium: 127.0.6533.73` * waited `~10s` and restarted `1.68.130 Chromium: 127.0.6533.73` once again * ensured that the following two parameters/switches are visible under `Command Line` via `brave://version` ``` --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed ``` **`Example`** | **`Example`** | **`Example`** | **`Example`** | **`Example`** ---------------|-----------------|----------------|----------------|------------------ ![IMG_0422](https://github.com/user-attachments/assets/d8736c42-b0ee-4bff-95c8-7afe7ae729dd) | ![IMG_0423](https://github.com/user-attachments/assets/e1bdc165-4ef1-4c20-b0ab-9adef594e5ed) | ![IMG_0424](https://github.com/user-attachments/assets/e05f0de0-d148-42bb-89ef-333f7e6ebfcb) | ![IMG_0425](https://github.com/user-attachments/assets/809aa608-b96d-42c0-865c-d84c3c281bba) | ![IMG_0426](https://github.com/user-attachments/assets/aba282dc-bf92-4df5-924c-4f9129186493) * ensured that the following parameters/switches are not visible/present via `Command Line` under `brave://version` ``` --accept-empty-variations-seed-signature --disable-variations-seed-fetch-throttling ``` **`Example`** | **`Example`** ---------------|---------------- ![IMG_0427](https://github.com/user-attachments/assets/3267515f-aa36-487d-aee5-78eb53cf7cc3) | ![IMG_0428](https://github.com/user-attachments/assets/0cbf9587-5991-43e1-8ff3-6ce441428d21)