search

brave / qa-resources

Mozilla Public License 2.0
21 stars 19 forks source link

Referral page change #153

Closed alexwykoff closed 4 years ago

alexwykoff commented 4 years ago

We have changed our referral page to redirect to brave.com/ with a parameter for the referral code.

We need QA to test this out and make sure its working correctly, taking into account things like account status on the part of the publisher as well as browser form factor.

The expected behavior for old urls is that they should redirect without issue. Please refer to the referral page spec as needed.

w0ts0n commented 4 years ago

Old setup: https://staging.brave.com/VZN173 New setup: https://staging.brave.com/?ref=VZN173

Test I think we should do:

LaurenWags commented 4 years ago

To expand upon the test plan from https://github.com/brave/qa-resources/issues/153#issuecomment-564106509, here's what was agreed to:

Desktop Check the desktop scenarios using the following types of pubs: a. A deleted referral code (provided by Cory) b. A specific pub state ref code (provided by Cory) c. A random invalid code (provided by Cory) d. A different pub state ref code (provided by Cory) e. A normal publisher (provided by Cory)

Check the following items on desktop for each of the publisher/creator states listed above using Brave, Chrome, Safari (macOS only), Edge (Windows only), Firefox.

  1. confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode
  2. confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode for Windows/macOS
  3. confirm after navigating around brave.com to various pages, your download is tagged with the refcode for Windows/macOS (need to check various download buttons/links on a variety of pages - staging.brave.com/features, staging.brave.com, staging.brave.com/download, etc)
  4. confirm no 404s shown when landing
  5. Unable to do this test for staging due to builds being downloaded from prod servers and ref codes used are staging - if you have access to publisher dashboard, confirm the referrals are counted on the pub dashboard in the Downloads Section

Mobile: Using Chrome on Android and Safari on iOS: Confirm use ref code -> get redirected

LaurenWags commented 4 years ago

macOS 10.13.6 x64 testing summary: Verified the following cases using Brave (1.1.20), Chrome (79.0.3945.79), Safari (12.1), Firefox (71.0)

a. Deleted referral code: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode (.pkg downloaded) PASS confirm after navigating around brave.com to various pages, your download is tagged with the refcode PASS confirm no 404s shown when landing

b. specific pub state PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode (.pkg downloaded) PASS confirm after navigating around brave.com to various pages, your download is tagged with the refcode PASS confirm no 404s shown when landing

c. random invalid code PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode (.pkg downloaded) PASS confirm after navigating around brave.com to various pages, your download is tagged with the refcode PASS confirm no 404s shown when landing

d. a different pub state PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode (.pkg downloaded) PASS confirm after navigating around brave.com to various pages, your download is tagged with the refcode PASS confirm no 404s shown when landing

e. normal publisher state PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm downloading from page immediately after redirect (no navigating away) downloads the installer tagged with refcode (.pkg downloaded) PASS confirm after navigating around brave.com to various pages, your download is tagged with the refcode PASS confirm no 404s shown when landing

Pages where dl button/link were checked: staging.brave.com/?ref=refcode (able to download refcode tagged installer bc this page is using a prod download link) staging.brave.com (unable to download installer due to staging download link, but dl buttons URL included refcode) staging.brave.com/download (unable to download installer due to staging download link, but dl button URLs included refcode) staging.brave.com/features (unable to download installer due to staging download link, but dl button URLs included refcode)

Pages which were not checked: staging.brave.com/tips/ staging.brave.com/download-beta staging.brave.com/download-dev staging.brave.com/download-nightly

btlechowski commented 4 years ago

Summery: all types of ref codes(user-funds-only, suspended publisher, invalid ref code, normal publisher, deleted ref code) behave the same from the point of view of the client

Important: It is impossible to test if proper installer is downloaded on staging, because the links on Download buttons points to staging.brave.com which aren't handled. Testing is needed on production server.

Tested on Chrome 78.0.3904.108, FF 72.0b4, Brave v1.1.20, IE 11 on Windows 7 x64; Microsoft Edge 44 on Windows 10 x64

KJO580 (user-funds-only)

NIC244 (suspended publisher)

DOG123 (invalid ref code)

LAU040 (normal publisher)

MIM486 (deleted ref code)

Issues found: Ref code not used on https://staging.brave.com/features/ on IE11

Note: Not a big deal, because the Download Brave button points to https://staging.brave.com/download which in turn is handled correctly

Steps:

  1. Open https://staging.brave.com/?ref=KJO580
  2. Open https://staging.brave.com/features/
  3. Scroll over Download Brave button

Actual behavior: Download Brave button points to https://staging.brave.com/download

Expected behavior: Download Brave button points to https://laptop-updates.staging.brave.com/download/KJO580

Note: https://staging.brave.com and https://staging.brave.com/download/ are handled correctly

LaurenWags commented 4 years ago

Android 5.1 on LG Nexus 5 testing summary: Used Chrome to verify the below items

a. Deleted referral code: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

b. specific pub state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

c. random invalid code: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

d. a different publisher state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

e. normal publisher state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

Pages where dl button were checked: staging.brave.com/?ref=refcode (unable to determine if refcode passed to GPS store as discussed) staging.brave.com (unable to download installer due to staging download link, but dl button URL included refcode) staging.brave.com/download (unable to download installer due to staging download link, but dl button URL included refcode)** staging.brave.com/features (unable to download installer due to staging download link, but dl button URL included refcode)

Pages which were not checked for download button/link: staging.brave.com/tips/ staging.brave.com/download-beta staging.brave.com/download-dev staging.brave.com/download-nightly

btlechowski commented 4 years ago

Tested on Chrome 78.0.3904.108 on Samsung s7 (ARM phone)

KJO580 (user-funds-only)

NIC244 (suspended publisher)

DOG123 (invalid ref code)

LAU040 (normal publisher)

MIM486 (deleted ref code)

LaurenWags commented 4 years ago

iPad 5th Gen using iOS 12.3.1 testing summary: Used Safari to verify the below items

a. Deleted referral code PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

b. specific publisher state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

c. random invalid code: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

d. different publisher state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

e. normal publisher state: PASS confirm staging.brave.com/refcode redirects to staging.brave.com/?ref=refcode PASS confirm Download button URL on page immediately after redirect (no navigating away) includes refcode PASS confirm after navigating around brave.com to various pages, your Download button URL includes refcode (note, see below for pages checked/not checked) PASS confirm no 404s shown when landing

Pages where dl button checked: staging.brave.com/?ref=refcode (unable to determine if refcode passed to App store as discussed) staging.brave.com (unable to download installer due to staging download link, but dl button URL included refcode) staging.brave.com/download (unable to download installer due to staging download link, but dl button URL included refcode) staging.brave.com/features (unable to download installer due to staging download link, but dl button URL included refcode)

Pages which were not checked for download URL: staging.brave.com/tips/ staging.brave.com/download-beta staging.brave.com/download-dev staging.brave.com/download-nightly

btlechowski commented 4 years ago

Tested on Chrome 78.0.3904.108 on Samsung Galaxy Tab S5e (Android 9.0) - ARM Tablet

KJO580 (user-funds-only)

NIC244 (suspended publisher)

DOG123 (invalid ref code)

LAU040 (normal publisher)

MIM486 (deleted ref code)

Issues found: Download Brave for Linux on Android tablet

Steps:

  1. Open https://staging.brave.com/?ref=LAU040
  2. Open https://staging.brave.com/download

Actual behavior: Download Brave for Linux button is shown

Expected behavior: Download Brave for Linux button is not shown

btlechowski commented 4 years ago

Testing summery on iPhone SE, iOS 13.3 LGTM

KJO580 (user-funds-only)

NIC244 (suspended publisher)

DOG123 (invalid ref code)

LAU040 (normal publisher)

MIM486 (deleted ref code)

LaurenWags commented 4 years ago

Code moved to Prod on Jan 2, 2020. Following checks were performed: macOS 10.14.6 macOS x64 - checked Chrome and Safari -

  1. Verified redirect occurred (ex. brave.com/ABC123 -> brave.com/?ref=ABC123)
  2. Verified after navigating around the site download buttons on brave.com/?ref=ABC123, brave.com, brave.com/download, and brave.com/features downloaded the referral promo branded pkg file.
  3. After downloading and installing pkg, confirmed promoCode in brave://local-state
  4. No 404s observed.

Android - LG Nexus 5 Android 5.1 - checked Chrome

  1. Verified redirect occurred (ex. brave.com/ABC123 -> brave.com/?ref=ABC123)
  2. Verified after navigating around the site download buttons on brave.com/?ref=ABC123, brave.com, brave.com/download, and brave.com/features contain the URL for https://laptop-updates.brave.com/download/ABC123
  3. Unable to confirm promoCode on Android
  4. No 404s observed.

iOS - iPad 5th Gen iOS 12.3.1 - checked Safari

  1. Verified redirect occurred (ex. brave.com/ABC123 -> brave.com/?ref=ABC123)
  2. Verified after navigating around the site download buttons on brave.com/?ref=ABC123, brave.com, brave.com/download, and brave.com/features contain the URL for https://laptop-updates.brave.com/download/ABC123
  3. Unable to confirm promoCode on iOS
  4. No 404s observed.
btlechowski commented 4 years ago

Windows 7 x64, Windows 7 x86 and Windows 10 x64 Used FF 72.0b10, Chrome 79.0.3945.88, IE. Verified promo_code in local state.

Android 8.0 on Samsung s7. Used Chrome

iOS - iPhone SE Used Safari