Manual test run on Win x64 for 1.31.x - Nightly (Checking C94)

kjozwiak commented 3 years ago

Installer

[x] Check the installer is close to the size of the last release
[x] Check signature: On macOS, run spctl --assess --verbose /Applications/Brave-Browser-Beta.app/ and make sure it returns accepted. On Windows, right-click on the brave_installer-x64.exe and go to Properties, go to the Digital Signatures tab and double-click on the signature. Make sure it says "The digital signature is OK" in the popup window

About pages

[x] Verify that both chrome:// and about:// forward to brave:// (run through several internal pages)

Importing

[x] Verify that you can import History, Favorites/Bookmarks and Passwords from Google Chrome
[x] Verify that you can import History, Favorites/Bookmarks, Passwords, Search Engines and Autofill from Firefox
[x] Verify that you can import Favorites/Bookmarks from Microsoft Edge
[x] Verify that importing bookmarks using Bookmark HTML File retains the folder structure on a clean profile

Context menus

[x] Verify you can block a page element using Block element via selector context-menu item
[x] Verify selecting Manage custom filters opens brave://adblock in a NTP
[x] Verify removing the rule from brave://adblock reflects the change on the website, after reload

Extensions/Plugins

[x] Verify pdfium, Torrent viewer extensions are installed automatically on fresh profile and cannot be disabled (they don't show up in brave://extensions)
[x] Verify older version of an extension gets updated to new version via Google server
[x] Verify older version of an extension gets updated to new version via Brave server
[x] Verify that magnet links and .torrent files correctly open WebTorrent and you're able to download the file(s)
- Tip: Free torrents available via https://webtorrent.io/free-torrents

Chrome Web Store (CWS)

[x] Verify that installing https://chrome.google.com/webstore/detail/adblock-plus-free-ad-bloc/cfhdojbkjhnklbpkdaibdccddilifddb from CWS displays the Brave has not reviewed the extension. warning via the "Add Extension" modal
[x] Verify that installing https://chrome.google.com/webstore/detail/lastpass-free-password-ma/hdokiejnpimakedhajhdlcegeplioahd from CWS doesn't display the Brave has not reviewed the extension. warning via the "Add Extension" modal

PDF

[x] Test that you can print a PDF
[x] Test that PDF is loaded over HTTPS at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf
[x] Test that PDF is loaded over HTTP at http://www.pdf995.com/samples/pdf.pdf
[x] Test that https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.6357&rep=rep1&type=pdf opens without issues

Widevine

[x] Verify Widevine Notification is shown when you visit Netflix for the first time
[x] Test that you can stream on Netflix on a fresh profile after installing Widevine
[x] Verify Widevine Notification is shown when you visit HBO Max for the first time
[x] Test that you can stream on HBO Max on a fresh profile after installing Widevine

Geolocation

[x] Check that https://browserleaks.com/geo works and shows correct location
[x] Check that https://html5demos.com/geo/ works but doesn't require an accurate location

Crash Reporting

[x] Check that loading brave://crash & brave://gpucrash causes the new tab to crash
[x] Check that brave://crashes lists the Uploaded Crash Report ID once the report has been submitted
[x] Verify the crash ID matches the report on Backtrace using _rxid equal [ value ]

Bravery settings

[x] Verify that HTTPS Everywhere works by loading http://https-everywhere.badssl.com/
[x] Turning HTTPS Everywhere off and Shields off both disable the redirect to https://https-everywhere.badssl.com/
[x] Verify that toggling Trackers & ads blocked works as expected
[x] Visit https://testsafebrowsing.appspot.com/s/phishing.html, verify that Safe Browsing (via our Proxy) works for all the listed items
[x] Visit https://www.blizzard.com and then turn on script blocking, page should not load.
[x] Test that 3rd party storage results are blank at https://jsfiddle.net/7ke9r14a/9/ when 3rd party cookies are blocked and not blank when 3rd party cookies are unblocked
[x] Test that https://mixed-script.badssl.com/ shows up as grey not red (no mixed content scripts are run)
[x] In brave://settings/security, choose a DNS provider from the providers listed under Use secure DNS, load https://browserleaks.com/dns, and verify your ISP's DNS resolvers aren't detected and shown; only your chosen DoH provider should appear.
[x] Open a New Private Window with Tor, load https://browserleaks.com/dns, and verify your ISP's DNS resolvers aren't detected and shown.

Fingerprint Tests

[x] Visit https://jsfiddle.net/bkf50r8v/13/, ensure 3 blocked items are listed in Shields. Result window should show got canvas fingerprint 0 and got webgl fingerprint 00
[x] Test that audio fingerprint is blocked at https://audiofingerprint.openwpm.com/ only when Block all fingerprinting protection is on
[x] Test that Brave browser isn't detected via user-agent, on https://www.whatismybrowser.com. The site will say "Looks like Brave on [Windows/Linux/macOS]", and yet, "But it's announcing that it's Chrome [version] on [platform]"
[x] Test that https://diafygi.github.io/webrtc-ips/ doesn't leak IP address for each option under Settings -> Privacy and Security -> WebRTC IP handling policy

Brave Ads

[x] Verify when you enable Rewards from panel or brave://rewards, Ads are enabled by default
[x] Verify Ads UI (panel, settings, etc) shows when in a region with Ads support
[x] Verify Ads UI (panel, settings, etc) does not show when in a region without Ads support. Verify the Ads panel does show the 'Sorry! Ads are not yet available in your region.' message.
[x] Verify when the system language is English, the Browser language is French, and you are in one of the supported regions, Ad notifications are still served to you.
[x] Verify you are served Ad notifications when Ads are enabled
- [x] Verify ad earnings are reflected in the rewards widget on the NTP.
[x] Verify when Ads are toggled off, there are no Ad messages in the logs
[x] Verify when Rewards are toggled off (but Ads were not explicitly toggled off), there are no Ads logs recorded
[x] Verify view/click/dismiss/landed ad notifications show in confirmations.json
[x] Verify pages you browse to are being classified in the logs
[x] Verify tokens are redeemed by viewing the logs (you can use --rewards=debug=true to shorten redemption time)
[x] Verify Ad is not shown if a tab is playing media and is only shown after it stops playing

Rewards

[x] Verify that none of the reward endpoints are being contacted when a user visits a media publisher (youtube.com, reddit.com, twitter.com, github.com) and hasn't interacted with rewards
- [x] Verify that rewards.brave.com, pcdn.brave.com, grant.rewards.brave.com or api.rewards.brave.com are not being contacted
[x] Verify you are able to create a new wallet.
[x] Verify you are able to restore a wallet.
[x] Verify account balance shows correct BAT and USD value.
[x] Verify actions taken (claiming grant, tipping, auto-contribute) display in wallet panel.
[x] Verify AC monthly budget shows correct BAT and USD value.
[x] Verify you are able to exclude a publisher from the auto-contribute table and popup list of sites.
[x] Verify you are able to exclude a publisher by using the toggle on the Rewards Panel.
[x] Verify you are able to perform an auto contribution.
- [x] Verify auto contribution is reflected in the rewards widget on the new-tab page (NTP).
[x] Verify monthly statement shows expected data.
[x] Verify when you click on the BR panel while on a site, the panel displays site-specific information (site favicon, domain, attention %).
[x] Verify BR panel shows message about an unverified publisher.
[x] Verify one time and monthly tip banners show a message about unverified publisher.
[x] Verify one time tip and monthly tip banners show a verified checkmark for a verified creator.
[x] Verify when you click on Send a tip, the custom tip banner displays if set up.
- [x] Verify custom tip banner is also displayed for monthly contribution.
[x] Verify you are able to make one-time tip and they display in Tips panel.
- [x] Verify tip is reflected in the rewards widget on the NTP.
- [x] Verify when you tip an unverified publisher, the one time tip is recorded in the Pending Contributions list.
[x] Verify you are able to make recurring tip and they display in Monthly Contributions panel.
- [x] Verify you are able to adjust your recurring tip amount from the BR panel.
- [x] Verify recurring tip is reflected in the rewards widget on the NTP.
[x] Verify you can tip a verified website.
- [x] Verify the website displays in the auto-contribute list after specified amount of time/visits per settings.
[x] Verify you can tip a verified YouTube creator.
- [x] Verify the YouTube creator displays in the auto-contribute list after specified amount of time/visits per settings.
[x] Verify you can tip a verified Vimeo creator.
- [x] Verify the Vimeo creator displays in the auto-contribute list after specified amount of time/visits per settings.
[x] Verify you can tip a verified Twitch creator.
- [x] Verify the Twitch creator displays in the auto-contribute list after specified amount of time/visits per settings.
[x] Verify you can tip a verified Twitter user from the panel.
[x] Verify you can tip a verified Twitter user via inline tip button.
- [x] Verify the in-line tip button is spaced properly.
[x] Verify you can tip a verified GitHub user from the panel.
[x] Verify you can tip a verified GitHub user via inline tip button.
- [x] Verify the GitHub creator displays in the auto-contribute list after specified amount of time/visits per settings.
[x] Verify you can tip a verified Reddit user from the panel.
[x] Verify you can tip a verified Reddit user via inline tip button.
[x] Verify if you disable auto-contribute you are still able to tip creators.
[x] Verify if auto-contribute is disabled AC does not occur.
[x] Verify if Rewards is disabled AC does not occur.
[x] Verify that disabling Rewards and enabling it again does not lose state.
[x] Adjust min visit/time in settings. Visit some sites to verify they are added to the table after the specified settings.
[x] Uphold cases
- [x] Verify you are able to connect a KYC'd Uphold wallet to Rewards.
- [x] Verify wallet balance in Brave updates when BAT is added to the Brave Browser card.
- [x] Verify if you only have user-controlled BAT (BAT in Uphold only), you can only tip KYC'd creators, any tips to non-KYC'd creators go to the Pending Contributions list.
- [x] Verify connected (verified but not KYC'd) publishers display messaging on panel and tip banner.
- [x] Verify you are able to perform an auto contribute using Uphold BAT.

Social-media blocking settings

[x] Verify individual Social media blocking buttons works as intended when enabled/disabled by visiting https://fmarier.github.io/brave-testing/social-widgets.html
[x] visit brave://settings/privacy -> Site and Shields Settings -> Cookies and site data and ensure that
- [x] both https://[*.]firebaseapp.com & https://accounts.google.com are added into Sites that can always use cookies when Allow Google login buttons on third party sites is enabled
- [x] both https://[*.]firebaseapp.com & https://accounts.google.com are removed from Sites that can always use cookies when Allow Google login buttons on third party sites is disabled
[x] ensure that you can log in into https://www.expensify.com while Allow Google login buttons on third party sites is enabled
[x] ensure that once Allow Google login buttons on third party sites has been disabled, you can't log in into https://www.expensify.com

Sync

[x] Verify you are able to create a sync chain and add a mobile/computer to the chain
[x] Verify you are able to join an existing sync chain using code words
[x] Verify the device name is shown properly when sync chain is created
[x] Verify you are able to add a new mobile device to the chain via QR code/code words
[x] Verify newly created bookmarks get sync'd to all devices on the sync chain
[x] Verify existing bookmarks on current profile gets sync'd to all devices on the sync chain
[x] Verify folder structure is retained after sync completes
[x] Verify bookmarks don't duplicate when sync'd from other devices
[x] Verify removing bookmark from device gets sync'd to all devices on the sync chain
[x] Verify adding/removing a bookmark in offline mode gets sync'd to all devices on the sync chain when device comes online
[x] With only two devices in chain, verify removing the other device resets the sync on b-c as well

Tor Tabs

[x] Visit https://check.torproject.org in a Tor window, ensure it shows a success message for using a Tor exit node
[x] Visit https://check.torproject.org in a Tor window, note down exit node IP address. Do a hard refresh (Ctrl+Shift+R/Cmd+Shift+R), ensure exit IP changes after page reloads
[x] Visit https://check.torproject.org in a Tor window, note down exit node IP address. Click New Tor connection for this site in app menu, ensure the exit node IP address changes after page is reloaded
[x] Visit https://protonirockerxow.onion & https://brave5t5rjjg3s6k.onion/ in a Tor window and ensure both pages resolve
[x] Visit https://browserleaks.com/geo in a Tor window, ensure location isn't shown
[x] Verify Torrent viewer doesn't load in a Tor window
[x] Ensure you are able to download a file in a Tor window. Verify all Download/Cancel, Download/Retry and Download works in Tor window

Cookie and Cache

[x] Go to http://samy.pl/evercookie/ and set an evercookie. Check that going to prefs, clearing site data and cache, and going back to the evercookie site does not remember the old evercookie value

Chromium/Brave GPU

[x] Verify that brave://gpu (Brave) matches chrome://gpu (Chrome) when using the same Chromium version

Startup & Components

[x] Verify that Brave is only contacting *.brave.com endpoints on first launch using either Charles Proxy, Fiddler, Wireshark or LittleSnitch (or a similar application)
[x] Remove the following component folders and ensure that they're being re-downloaded after restarting the browser:
- [x] afalakplffnnnlkncjhbmahjfjhmlkal: AutoplayWhitelist.dat, ExtensionWhitelist.dat, ReferrerWhitelist.json and Greaselion.json
- [x] CertificateRevocation
- [x] cffkpbalmllkdoenhmdmpbkajipdjfam: rs-ABPFilterParserData.dat & regional_catalog.json (AdBlock)
- [x] gccbbckogglekeggclmmekihdgdpdgoe: (Sponsored New Tab Images)
- [x] jicbkmdloagakknpihibphagfckhjdih: speedreader-updater.dat
- [x] oofiananboodjbbmdelgdommihjbkfag: HTTPSE
- [x] Safe Browsing
[x] Restart the browser, load brave://components, wait for 8 mins and verify that no component shows any errors

Note: Always double check brave://components to make sure there's no errors/missing version numbers

Session storage

[x] Temporarily move away your browser profile and test that a new profile is created on browser relaunch
- macOS - ~/Library/Application\ Support/BraveSoftware/
- Windows - %userprofile%\appdata\Local\BraveSoftware\
- Linux(Ubuntu) - ~/.config/BraveSoftware/
[x] Test that both windows and tabs are being restored, including the current active tab
[x] Ensure that tabs are being lazy loaded when a previous session is being restored

Upgrade

[x] Make sure that data from the last version appears in the new version OK
[x] Ensure that brave://version lists the expected Brave & Chromium versions
[x] With data from the last version, verify that:
- [x] Bookmarks on the bookmark toolbar and bookmark folders can be opened
- [x] Cookies are preserved
- [x] Installed extensions are retained and work correctly
- [x] Opened tabs can be reloaded
- [x] Stored passwords are preserved
- [x] Sync chain created in previous version is retained
- [x] Social media-blocking buttons changes are retained
- [x] Rewards
- [x] Wallet balance is retained
- [x] Auto-contribute list is retained
- [x] Both Tips and Monthly Contributions are retained
- [x] Wallet panel transactions list is retained
- [x] Changes to rewards settings are retained
- [x] Ensure that Auto Contribute is not being enabled when upgrading to a new version if AC was disabled
- [x] Ads
- [x] Both Estimated pending rewards & Ad notifications received this month are retained
- [x] Changes to ads settings are retained
- [x] Ensure that ads are not being enabled when upgrading to a new version if they were disabled
- [x] Ensure that ads are not disabled when upgrading to a new version if they were enabled

Hardware Wallet Test (To be checked on each major CR bump)

[x] Verify you can link hardware wallet using Trezor and unlock the wallet
[x] Verify you can link hardware wallet using Ledger Nano and unlock the wallet
[x] Verify you can perform a transaction with hardware wallet using Ledger Nano or Trezor

kjozwiak commented 3 years ago

Compared brave://components against 1.29.81 Chromium: 93.0.4577.82 which is the current release and the only difference is that Client Side Phishing Detection is missing due to https://github.com/brave/brave-browser/issues/16379 landing in 1.30.x.

Following component was removed from 1.30.x and later:

Also ensured that visiting brave://settings/content/autoplay, brave://settings/content/siteDetails?site=https%3A%2F%2Fbrave.com or brave://settings/content/ethereum doesn't crash the browser like it did with the c93 regression as per https://github.com/brave/brave-browser/issues/17808.

kjozwiak commented 3 years ago

Also verified the sections that were mentioned by @mkarolin via https://github.com/brave/brave-browser/issues/17170#issue-953064441 on Win 10 x64 using the following build:

Brave | 1.31.44 Chromium: 94.0.4606.41 (Official Build) nightly (64-bit)
-- | --
Revision | 333e85df3c9b656b518b5f1add5ff246365b6c24-refs/branch-heads/4606@{#845}
OS | Windows 10 OS Version 2009 (Build 19042.1202)

[x] Profile picker appearance
[x] Permission lifetime bubbles
[x] CC import from Chrome profiles
[x] Windows: upgrade from prior version, test for all channels

We'll run through the following once the above is completed via macOS. Unfortunately, there was a regression with the Brave Safe Storage via https://github.com/brave/brave-browser/issues/18138 which is being fixed by https://github.com/brave/brave-core/pull/10108.

MacOS: ad notifications
MacOS: saved passwords accessible after upgrade

GeetaSarvadnya commented 3 years ago

Compared the CR93 and CR94 brave://settings. Gone through all the settings one by one and confirmed settings works as expected. New settings added as part of CR94 are listed below: image (28) image (27)

brave / brave-browser