Manual test run on Linux for 1.58.x - Release

LaurenWags commented 1 year ago

Installer

[x] Check the installer is close to the size of the last release
[ ] Check signature:
- [ ] If macOS, using x64 binary run spctl --assess --verbose for the installed version and make sure it returns accepted
- [ ] If macOS, using universal binary run spctl --assess --verbose for the installed version and make sure it returns accepted
- [ ] If Windows right click on the brave_installer-x64.exe and go to Properties, go to the Digital Signatures tab and double click on the signature. Make sure it says "The digital signature is OK" in the popup window

About pages

[x] Verify that both chrome:// and about:// forward to brave:// (run through several internal pages)

Adblock lists and filters

[x] Navigate to brave://adblock and confirm you are redirected to brave://settings/shields/filters
[x] Filter lists
- [x] Type "cookie" into the "Filter lists" search box, and verify that the list is filtered down to just the Easylist-Cookie List - Filter Obtrusive Cookie Notices
- [x] Enable the Easylist-Cookie List and refresh the page
- [x] Expand the regional filter lists and verify that the cookie list is still enabled
[x] Add custom filter lists
- [x] Enter https://raw.githubusercontent.com/ryanbr/fanboy-adblock/master/fanboy-antifonts.txt into Enter filter list URL and press Add
- [x] Ensure that the Custom lists table appears, with the newly added entry; Last updated should be now (e.g. 0 seconds ago)
- [x] Refresh the page after a few seconds and ensure that the Last updated time reflects the time that has passed
- [x] Use the dropdown next to the entry and verify that all options (Update now, View source, Unsubscribe) are shown
  - [x] Pressing the Update now button should update the timestamp appropriately
  - [x] View source should open the text of the filter list
  - [x] Unsubscribe should delete the entry from the list and hide the table once again
[x] Create custom filters
- [x] The Create custom filters section should be visible with a text box and Save changes button
- [x] Adding text to the box, pressing Save changes, and refreshing the page should result in the same text appearing in the box after the reload

Importing

[x] Verify that you can import History, Favorites/Bookmarks and Passwords from Google Chrome
[x] Verify that you can import History, Favorites/Bookmarks, Passwords, Search Engines and Autofill from Firefox
[x] Verify that you can import Favorites/Bookmarks from Microsoft Edge
[x] Verify that importing bookmarks using Bookmark HTML File retains the folder structure on a clean profile

Context menus

[x] Verify you can block a page element using Block element via selector context-menu item
[x] Verify selecting Manage custom filters opens brave://adblock in a NTP
[x] Verify removing the rule from brave://adblock reflects the change on the website, after reload

Extensions/Plugins

[x] Verify pdfium, Torrent viewer extensions are installed automatically on fresh profile and cannot be disabled (they don't show up in brave://extensions)
[x] Verify older version of an extension gets updated to new version via Google server
[x] Verify that magnet links and .torrent files correctly open WebTorrent and you're able to download the file(s)
- Tip: Free torrents available via https://webtorrent.io/free-torrents

IPFS

[x] Load vitalik.eth. Click on Proceed using Infura server. Click on Use a Brave local IPFS node. Verify the content loads from ipfs://bafybeigsn4u4nv4uyskxhewakk5m2j2lluzhsbsayp76zh7nbqznrxwm7e/.

Chrome Web Store (CWS)

[x] Verify that installing https://chrome.google.com/webstore/detail/adblock-plus-free-ad-bloc/cfhdojbkjhnklbpkdaibdccddilifddb from CWS displays the Brave has not reviewed the extension. warning via the "Add Extension" modal
[x] Verify that installing https://chrome.google.com/webstore/detail/lastpass-free-password-ma/hdokiejnpimakedhajhdlcegeplioahd from CWS doesn't display the Brave has not reviewed the extension. warning via the "Add Extension" modal

PDF

[x] Test that you can print a PDF
[x] Test that PDF is loaded over HTTPS at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf
[x] Test that PDF is loaded over HTTP at http://www.pdf995.com/samples/pdf.pdf

Widevine

[x] Verify Widevine Notification is shown when you visit Netflix for the first time
[x] Test that you can stream on Netflix on a fresh profile after installing Widevine
[x] Verify Widevine Notification is shown when you visit HBO Max for the first time
[x] Test that you can stream on HBO Max on a fresh profile after installing Widevine
[ ] If macOS, run the above Widevine tests for both x64 and universal builds

Geolocation

[x] Check that https://browserleaks.com/geo works and shows correct location
[x] Check that https://html5demos.com/geo/ works but doesn't require an accurate location

Crash Reporting

[x] Check that loading brave://crash & brave://gpucrash causes the new tab to crash
[x] Check that brave://crashes lists the Uploaded Crash Report ID once the report has been submitted
[x] Verify the crash ID matches the report on Backtrace using _rxid equal [ value ]

Bravery settings

[x] Verify that HTTPS Everywhere works by loading http://https-everywhere.badssl.com/
[x] Turning HTTPS Everywhere off and Shields off both disable the redirect to https://https-everywhere.badssl.com/
[x] Verify that toggling Trackers & ads blocked works as expected
[x] Visit https://testsafebrowsing.appspot.com/s/phishing.html, verify that Safe Browsing (via our Proxy) works for all the listed items
[x] Visit https://www.blizzard.com and then turn on script blocking, page should not load.
[x] Test that 3rd party storage results are blank at https://jsfiddle.net/7ke9r14a/9/ when 3rd party cookies are blocked and not blank when 3rd party cookies are unblocked
[x] Test that https://mixed-script.badssl.com/ shows up as grey not red (no mixed content scripts are run)
[x] In brave://settings/security, choose a DNS provider from the providers listed under Use secure DNS, load https://browserleaks.com/dns, and verify your ISP's DNS resolvers aren't detected and shown; only your chosen DoH provider should appear.
[x] Open a New Private Window with Tor, load https://browserleaks.com/dns, and verify your ISP's DNS resolvers aren't detected and shown.

TLS Pinning

[x] Visit https://ssl-pinning.someblog.org/ and verify a pinning error is displayed
[x] Visit https://pinning-test.badssl.com/ and verify a pinning error is not displayed

Fingerprint Tests

[x] Test that https://diafygi.github.io/webrtc-ips/ doesn't leak IP address for each option under Settings -> Privacy and Security -> WebRTC IP handling policy

Brave Ads

[x] Verify when you enable Rewards from panel or brave://rewards, Ads are enabled by default
[x] Verify Ads UI (panel, settings, etc) shows when in a region with Ads support
[x] Verify Ads UI (panel, settings, etc) does not show when in a region without Ads support. Verify the Ads panel does show the 'Sorry! Ads are not yet available in your region.' message.
[x] Verify when the system language is English, the Browser language is French, and you are in one of the supported regions, Ad notifications are still served to you.
[x] Verify you are served Ad notifications when Ads are enabled
- [x] Verify ad earnings are reflected in the rewards widget on the NTP.
[x] Verify when Ads are toggled off, there are no Ad messages in the logs
[x] Verify when Rewards are toggled off (but Ads were not explicitly toggled off), there are no Ads logs recorded
[x] Verify view/click/dismiss/landed ad notifications show in confirmations.json
[x] Verify pages you browse to are being classified in the logs
[x] Verify tokens are redeemed by viewing the logs (you can use --rewards=debug=true to shorten redemption time)
[x] Verify Ad is not shown if a tab is playing media and is only shown after it stops playing

Rewards

[x] Verify that none of the reward endpoints are being contacted when a user visits a media creator (youtube.com, reddit.com, twitter.com, github.com) and hasn't joined rewards
- [x] Verify that rewards.brave.com, pcdn.brave.com, grant.rewards.brave.com or api.rewards.brave.com are not being contacted
[x] Verify you are able to create a new Rewards profile and are in the unverified state by default
- [x] Verify when you visit a creator in this state the panel shows a prompt to connect a custodian and no BAT information (earnings, balance, etc) is displayed
- [x] Verify when you click on an in-line tip button in this state, the tip banner indicates you must connect a custodian
- [x] Verify when you view the NTP widget in this state it shows a prompt to connect a custodian and no BAT information (earnings, balance, etc) is displayed
- [x] Verify when you visit brave://rewards in this state there are no Auto Contribution, Tipping, or summary panels on this page
- [x] Verify you can toggle ads off/on from this page and the panel and NTP widget respond accordingly
- [x] Verify you can toggle rewards off/on from the panel and the brave://rewards page and NTP widget respond accordingly
[x] Verify you are able to connect a custodian
- [x] Verify Rewards balance shows correct BAT and USD value on brave://rewards, panel, and NTP widget after you connect
- [x] Verify when you click on the BR panel while on a site, the panel displays site-specific information (site favicon, domain, attention %)
- [x] Verify BR panel and tip banner shows message about an unverified creator
- [x] Verify BR panel and tip banner shows message about a creator with a different custodian
- [x] Verify you are able to tip a creator who has the same custodian as you
- [x] Verify you are able to perform an auto contribution
- [x] Verify if you disable auto-contribute you are still able to tip creators
- [x] Verify if auto-contribute is disabled AC does not occur
- [x] Verify that there is no "Disconnect" option once connected to a custodian
[x] Verify that you are able to reset rewards
- [x] Verify after rewards are reset, the panel, brave://rewards page, and NTP widget to back to the unverified state

Social-media blocking settings

[x] Verify individual Social media blocking buttons works as intended when enabled/disabled by visiting https://fmarier.github.io/brave-testing/social-widgets.html
[x] visit brave://settings/privacy -> Site and Shields Settings -> Cookies and site data and ensure that
- [x] both https://[*.]firebaseapp.com & https://accounts.google.com are added into Sites that can always use cookies when Allow Google login buttons on third party sites is enabled
- [x] both https://[*.]firebaseapp.com & https://accounts.google.com are removed from Sites that can always use cookies when Allow Google login buttons on third party sites is disabled
[x] ensure that you can log in into https://www.expensify.com while Allow use of third-party cookies for legacy Google Sign-In is enabled
[x] ensure that once Allow use of third-party cookies for legacy Google Sign-In has been disabled, you can't log in into https://www.expensify.com

Sync

[x] Verify you are able to create a sync chain and add a mobile/computer to the chain
[x] Verify you are able to join an existing sync chain using code words
[x] Verify the device name is shown properly when sync chain is created
[x] Verify you are able to add a new mobile device to the chain via QR code/code words
[x] Verify newly created bookmarks get sync'd to all devices on the sync chain
[x] Verify existing bookmarks on current profile gets sync'd to all devices on the sync chain
[x] Verify folder structure is retained after sync completes
[x] Verify bookmarks don't duplicate when sync'd from other devices
[x] Verify removing bookmark from device gets sync'd to all devices on the sync chain
[x] Verify adding/removing a bookmark in offline mode gets sync'd to all devices on the sync chain when device comes online
[x] With only two devices in chain, verify removing the other device resets the sync on b-c as well

Tor Tabs

[x] Visit https://check.torproject.org in a Tor window, ensure it shows a success message for using a Tor exit node
[x] Visit https://check.torproject.org in a Tor window, note down exit node IP address. Do a hard refresh (Ctrl+Shift+R/Cmd+Shift+R), ensure exit IP changes after page reloads
[x] Visit https://check.torproject.org in a Tor window, note down exit node IP address. Click New Tor connection for this site in app menu, ensure the exit node IP address changes after page is reloaded
[x] Visit https://protonmailrmez3lotccipshtkleegetolb73fuirgj7r4o4vfu7ozyd.onion, https://brave4u7jddbv7cyviptqjc7jusxh72uik7zt6adtckl5f4nwy2v72qd.onion/, and https://search.brave4u7jddbv7cyviptqjc7jusxh72uik7zt6adtckl5f4nwy2v72qd.onion/ in a Tor window and ensure all pages resolve
[x] Visit https://browserleaks.com/geo in a Tor window, ensure location isn't shown
[x] Verify Torrent viewer doesn't load in a Tor window
[x] Ensure you are able to download a file in a Tor window. Verify all Download/Cancel, Download/Retry and Download works in Tor window

Cookie and Cache

[x] Go to http://samy.pl/evercookie/ and set an evercookie. Check that going to prefs, clearing site data and cache, and going back to the evercookie site does not remember the old evercookie value

Chromium/Brave GPU

[x] Verify that brave://gpu (Brave) matches chrome://gpu (Chrome) when using the same Chromium version

Startup & Components

[x] Verify that Brave is only contacting *.brave.com endpoints on first launch using either Charles Proxy, Fiddler, Wireshark or LittleSnitch (or a similar application)
- [x] Verify that opening a NTP doesn't trigger any outbound connections related to widgets without user interaction
[x] Remove the following component folders and ensure that they're being re-downloaded after restarting the browser:
- [x] afalakplffnnnlkncjhbmahjfjhmlkal: AutoplayWhitelist.dat, ExtensionWhitelist.dat, ReferrerWhitelist.json and Greaselion.json
- [x] CertificateRevocation
- [x] cffkpbalmllkdoenhmdmpbkajipdjfam: rs-ABPFilterParserData.dat & regional_catalog.json (AdBlock)
- [x] gccbbckogglekeggclmmekihdgdpdgoe: (Sponsored New Tab Images)
- [x] jicbkmdloagakknpihibphagfckhjdih: speedreader-updater.dat
- [x] oofiananboodjbbmdelgdommihjbkfag: HTTPSE
- [x] Safe Browsing
[x] Restart the browser, load brave://components, wait for 8 mins and verify that no component shows any errors

Note: Always double check brave://components to make sure there's no errors/missing version numbers

Session storage

[x] Temporarily move away your browser profile and test that a new profile is created on browser relaunch
- macOS - ~/Library/Application\ Support/BraveSoftware/
- Windows - %userprofile%\appdata\Local\BraveSoftware\
- Linux(Ubuntu) - ~/.config/BraveSoftware/
[x] Test that both windows and tabs are being restored, including the current active tab
[x] Ensure that tabs are being lazy loaded when a previous session is being restored

Upgrade

[x] Make sure that data from the last version appears in the new version OK
[x] Ensure that brave://version lists the expected Brave & Chromium versions
[x] With data from the last version, verify that:
- [x] Bookmarks on the bookmark toolbar and bookmark folders can be opened
- [x] Cookies are preserved
- [x] Installed extensions are retained and work correctly
- [x] Opened tabs can be reloaded
- [x] Stored passwords are preserved
- [x] Sync chain created in previous version is retained
- [x] Social media-blocking buttons changes are retained
- [x] Custom filters under brave://settings/shields/filters are retained
- [x] Custom lists under brave://settings/shields/filters are retained
- [x] Rewards
- [x] BAT balance is retained
- [x] Auto-contribute list is retained
- [x] Both Tips and Monthly Contributions are retained
- [x] Panel transactions list is retained
- [x] Changes to rewards settings are retained
- [x] Ensure that Auto Contribute is not being enabled when upgrading to a new version if AC was disabled
- [x] Ads
- [x] Both Estimated pending rewards & Ad notifications received this month are retained
- [x] Changes to ads settings are retained
- [x] Ensure that ads are not being enabled when upgrading to a new version if they were disabled
- [x] Ensure that ads are not disabled when upgrading to a new version if they were enabled

MadhaviSeelam commented 1 year ago

Verification PASSED using

Brave | 1.58.122 Chromium: 117.0.5938.62 (Official Build) (64-bit)
-- | --
Revision | 623437a8c792953b7bc4574f2d9ccdfd59e0ca67
OS | Linux

Verified following: Installer -->> Fingerprint tests Rewards & Upgrade

stephendonner commented 1 year ago

Verification `PASSED` using

Brave | 1.58.122 Chromium: 117.0.5938.62 (Official Build) (64-bit)
-- | --
Revision | 623437a8c792953b7bc4574f2d9ccdfd59e0ca67
OS | Linux

Verified the following sections:

Social-media blocking settings through Session storage

btlechowski commented 1 year ago

Verification PASSED

Brave	1.58.122 Chromium: 117.0.5938.62 (Official Build) (64-bit)
Revision	623437a8c792953b7bc4574f2d9ccdfd59e0ca67
OS	Linux

Verified Brave Ads

LaurenWags commented 1 year ago

Closing off per the above comments from @MadhaviSeelam @stephendonner @btlechowski - lmk if any concerns 👍🏻

penguine0815 commented 1 year ago

@LaurenWags Can you please look at comment #33086. I am writing to you because I am not getting any feedback and I don't know if this is already being worked on. I appreciate any comment/help to use the latest Brave version. Thank you very much and regards.

brave / brave-browser