Tests sets reports v3.11

mneunomne commented 3 years ago

Documenting here what has already been tested

mneunomne commented 3 years ago

Menu

[x] Check all buttons and information (including the number of Ads and $$ cost, hidden if no ads)
[x] Hover over Ad images and check zooming
[x] Check that menu is correctly updated after ad-visiting
[x] Check ad-visiting animation
[x] Check there is no text overlap in the menu for text ads
[x] Go to extensions page, the menu should show "AdNauseam is paused on current pages", make sure that the layout is correct.
[x] Check that the Adnauseam logo links to homepage
[x] Check that the uBlock menu is properly displayed and everything is readable

Vault

[x] Verify loading animation, and centering/scaling of Ads on load
[x] Zoom in/out using mouse wheel and buttons in the top-left corner
[x] Check function/layout of Inspector (for both image and text ads)
[x] Check correct function of the date-filter slider at the bottom
[x] Check that AdNauseam logo in the bottom-right corner links correctly to homepage
[x] Check correct calculation of ad cost
[x] Test import/export/clear functions
[x] Right-click on an ad and delete it. Refresh the page and check that it is successfully deleted
[x] Check the statistics panel and make sure that an ad is correctly categorized for ad types, ad networks and sites
[x] Click on "My statistics", the panel should be minimized with smooth animation

Dashboard

[x] Check layout of buttons, check-boxes, and links in each tab
[x] Test that all (i)nfo buttons lead to correct FAQ pages in each tab
[x] On settings tab (options.html) check that when clicking or hiding are disabled, then all of their sub-items are also disabled
[x] On settings tab (options.html), change click probabilities and refresh the page, make sure that the probabilities is set to the new value after page refresh
[x] Scroll to the bottom of the settings tab, test reset/backup/restore buttons
[x] Click on a filter in 3p-filters and make sure that asset viewer works
[x] Test all the import/export/apply buttons for "My Filters"
[x] Test that import/export buttons work for "My Rules"
[x] Test creating deleting and modifying the rules in "My Rules" (make sure to test strictBlock which is AdNauseam-specific
[x] Test whitelist
[ ] Check 2 version numbers in 'About' page (links.html)

Toolbar Icon

[x] Check that the icon in the toolbar is in purple when AdNauseam is active
[x] Check that the icon in the toolbar is in grey when AdNauseam is disabled
[x] Check that the icon in the toolbar is in green when on DNT pages
[x] Icon should animate each time an Ad is clicked
[x] Double check that the badge number is correctly updated
[x] Check that the text on hover is correct for active and disabled state

Notifications

[x] Check that disabling hiding, clicking, or blocking on the first-run generates a notification warning at the top of the dashboard (all tabs), menu, and vault page (after first-run is closed or 'ok' is clicked).
[x] Check that disabling hiding, clicking, or blocking on the dashboard/options page generates a notification warning at the top of the dashboard(all tabs), menu, and vault page.
[x] Check that disabling EasyList/AdNauseam filters on the dashboard/3p-filters page generates a notification warning at the top of the dashboard(all tabs), menu, and vault page.
[x] Check that clicking 'reactivate' on a notification warning at the top of the dashboard, menu, or vault pages causes the visual state to change correctly, and the setting in question to be updated correctly.
[x] In Firefox, choose "strict" for Enhanced Tracking Protection in about:preferences#privacy. Check that a notification shows up after the setting is changed.
[x] Check that the vault (if open) is updated when notifications change
[x] Check that the dashboard (if open) is updated when notifications change
[x] Dashboard: settings (options.html) Check that page refreshes correctly when a notification is 'reactivated'
[x] Dashboard::3rd-Party (3p-filters.html) Check that page refreshes correctly when a list-notification is 'reactivated'
[ ] [Opera only] If "allow access to search page results" is unchecked, AdNauseam will show a corresponding notification in menu on the search result page. (Test Google search result for "credit card")

Logger

[x] Visit a random web page then open the logger
[x] Check that the logger is properly displayed and all entries are readable
[x] Click on the 'i' icon on the top right corner of the screen, make sure that it links to AdNauseam's wiki page for the logger

Ad Hiding/Parsing

[x] Test a set of pages, and make sure that ads are being captured and visited
- TextAds: visit Google/Ask/Yahoo/Bing/DuckDuckGo to search for keywords such as "credit card" or "loan". tested example
- ImageAds: visit NYTimes/Youtube/Facebook(Right column only)... and verify image ads are hidden and appear in menu
[x] Test Ad-parsing from within dynamically-created iFrames: hidden and collected
Test cases: Dynamic_iframe
[x] Test Ad-parsing from cosmetically-filtered iFrames: hidden but not collected (without cosmetic filters for any contained elements) here

Ad clicking

Visual
Check whether the clicking status of an ad is correctly indicated by the color of its border:

[x] Blue - Pending
[x] Purple - Clicked
[x] Red - Failed
[x] Green - DNT

Ad Visits

[x] Test different click probabilities

Text

[x] If an ad is not visited because of the clicking frequency, the ad should be marked as "SKIPPED: Click Frequency "
[x] If an ad is not visited because the user has already clicked it, the ad should be marked as "SKIPPED: Clicked by user "
[x] When an ad is collected from a dnt site (Ex:duckduckgo), the ad should be marked as "SKIPPED: DNT Allowed" ("Allowed via Do Not Track Policy" in vault)
[x] When clicking is disabled, all ads that are not "visited" or "failed" should be marked as "SKIPPED: Clicking Disabled:"

Real-world test cases for ad collection and clicking

mneunomne commented 3 years ago

Other AdBlockers

[x] Install uBlock Origin
[x] Remove and reinstall AdNauseam to trigger the first-run page
[x] check if there is a warning about other ad blockers on the first-run page,menu and vault
[x] [chromium based only] click the 'disable' button on the warning message, it should lead to the extension page
[x] [chromium based only] disable uBlock and check whether the message is gone in vault and menu. (If the page was opened, it need to be refreshed)
[x] Test this for Adblock Plus/AdBlock/uBlock as well

Blocking

[x] Check that the request from the following test page is blocked by adnauseam by using the uBlock logger here

Redirect-rules

[x] Check that the "redirect" and "redirect-rule" are properly working, comparing behaviour with uBlock Origin. Example of website with both types of rules here. For further information follow the discussion on this issue https://github.com/dhowe/AdNauseam/issues/1771#issuecomment-807294379

New Features

Strict Blocking

[x] Enable global strict blocking in settings, open the logger and go to sites such as nytimes. All blocking rules(including easyList) should be blocked(in red).
[x] Go to "My rules" and add * doubleclick.net * strictBlock; Commit the rule and make sure that nothing turns red in codeMIrror; Go to nytimes and make sure that requests from doubleclick.net are all blocked by easylist.
[x] Go to "My rules" and add nytimes.com * * strictBlock; Commit the rule and make sure that nothing turns red in codeMIrror; Go to nytimes and make sure that blocking filters from easyList are triggered.
[x] Go to "My rules" and add nytimes.com * image strictBlock; Commit the rule and make sure that nothing turns red in codeMIrror; Go to nytimes and make sure that blocking filters on images are triggered from easyList, but other types of requests are still allowed.

mneunomne commented 3 years ago

DoNotTrack (DNT)

DNT: Go to a DNT page (Ex: duckduckgo.com) and check the following items:

Headers/Cookies

To check ad visits headers, click on request in browser's network panel, select 'Headers' (For a more detailed network logging, you can check Chrome Net-Internals of the tools listed below.)

[x] DNT-header is being sent correctly for ALL requests (including Ad visits) if either disableClickingForDNT and disableHidingForDNT are enabled (and vice versa)
[x] DNT Cookies are allowed (v3.1 and later)

Hiding/Clicking

[x] Ads are visible when domain is 1st-party if disableHidingForDNT is checked
[x] Requests are not blocked when domain is 1st-party iff disableHidingForDNT is checked
[x] Requests are not blocked when domain is 3rd-party iff disableHidingForDNT is checked(here)
[x] No clicking when ad domain is on list iff disableClickingForDNT is checked (especially for 3rd-party)
[x] If both DNT options are disabled, then hiding, blocking, visiting occur again as usual(both first party and 3rd-party)

Interface

[x] DNT list at top of whitelist.html is checked when either DNT option is checked, otherwise unchecked
[x] Check that changing the two DNT settings will trigger DNT notifications with corresponding messages in the menu(dnt sites only)
[x] Check the layout of each DNT notification("?" should be in the middle, 'settings' button appears on hover)

Test Page for 3rd-party DNT

mneunomne commented 3 years ago

@dhowe can you help with these tests?

Cookies

[x] Test that no cookies are accepted from adn-ALLOWed requests:
- Make sure ‘Activate debugging mode’ is set to true to view log info in the console
- Visit sites(nytimes for example) where one or more 3rd-party requests are adn-ALLOWed and note their domains
[x] Test that outgoing cookies are always sent
[x] Test that incoming cookies are not allowed from responses to Ad visits, by checking cookies before/after (if only Channel ID cookies are allowed, it is ok for now) TestCase
To check headers for ad visits, please go to developer tools of the browser -> Network -> click on a request -> select 'Headers'. You can also click on 'XHR' type to filter the requests.(For a more detailed network logging, you can check Chrome Net-Internals of the tools listed below.)Please notice that, sometimes there is a “CAUTION: provisional headers are shown” message when you check headers in the console. If you see this message, the headers shown are not reliable. Therefore it should not be considered as a reference when checking the headers during test release/ debugging. Check other requests or use Chrome Net-Internals instead.

dhowe / AdNauseam