Blacklist for Auto-Save

[s-crypt]

Is your feature request related to a problem/ issue? Please describe.

• There needs to be a blacklist to exclude auto saving sites. Private URLs and searches can spam the WBM.

Describe the solution you'd like

• An implemented blacklist that has both defaults (like search engines. There is already a basic list here and user added domains to not auto archive.

Describe alternatives you've considered

• Besides doing everything manually and/or turning on private mode, there is no other option.

Additional context

• It might also be a good idea to not archive the Alexa top 100-500 because they are already pretty much covered by manual archives, as well as pattern matching a url for a search query.

Edit: Some things that should be blacklisted:

• Anything with search?, ?s, s?, ?searchterm, ?search, ?q, ?query, search.cfm, search?q, result?, ?st, search.aspx?, */search/*, *.amazon.com/s/, search#, ?f, ?t - used for search terms, can still be manually archived
• Delete anything after ?utm_source - used for attribution. Remove anything after to get just the pure URL
• Delete anything after # - used for document locations

[cgorringe]

This is a good point, particularly while using search engines and the ?utm_source or that one that facebook uses.

[s-crypt]

The ClearURL extension project actually has a list of regex matches to detect and make clean URLs. https://docs.clearurls.xyz/latest/specs/rules/ https://rules2.clearurls.xyz/data.minify.json

[cgorringe]

@s-crypt Thanks for the info!

[cgorringe]

@s-crypt Really appreciate the suggestion and info on ClearURLs! I've created a new feature request at #722 - Thanks!

[cgorringe]

Adding some notes on terminology for future reference:

• avoid: black / white.
• use: exclude, block, reject, minus.
• use: include, allow, permit, plus, safe.

"whitelist to exclude auto saving sites" actually sounds like the opposite. Maybe we can call this an "Exclude URL Patterns" list of regex's, or how about "Filtered URL Patterns".

[s-crypt]

Sorry, I mixed up blacklist and whitelist when I originally wrote this

[SpongebobSquamirez]

I WANT THIS!!!

[SpongebobSquamirez]

Forgot I already posted on this, but just coming back again to see if this feature has been implemented (guess not) and to reiterate that I really want this, for things like gmail and facebook especially.

[cgorringe]

Finally implemented this in PR #845 if anyone wants to test it out. I should probably add some more URL patterns to the Default List so I welcome any suggestions!

[contextnerror]

Is there a way to have an "Include List"? That is, only save a URL if it matches the user-defined list?

[cgorringe]

Hi @contextnerror, although possible, I'd think it wouldn't be as necessary since you can always explicitly tap Save Page Now.

We used to have a Bulk Save feature where you could import your bookmarks. This could be an alternative as a bookmark list is like an "include list", although it would take some extra steps. The Bulk Save feature has been temporarily removed but we plan to bring it back in the future.

[DesertStars]

Hi @contextnerror, although possible, I'd think it wouldn't be as necessary since you can always explicitly tap Save Page Now.

We used to have a Bulk Save feature where you could import your bookmarks. This could be an alternative as a bookmark list is like an "include list", although it would take some extra steps. The Bulk Save feature has been temporarily removed but we plan to bring it back in the future.

While I get that "Save Page Now" is useful to quickly archive a webpage, it's still a bit of chore to do that for every link that hasn't been archived so I think adding an "Include List" as an option would be extremely helpful especially when it's used together "Auto Save Page".

To explain my point further, I believe the way "Auto Save Page" works right now is rather inconvenient because if you turned it on, you'd have to go and add all kind of urls to your "Exclude List" to stop private webpages that'd be pointless or even harmful to have it saved on the Wayback Machine. On other hand, a whitelist would be much more logical since it would require minimal action from the user and there would be very low risk of spamming the Wayback Machine compared to a blacklist.

For example, I've been using the extension for a while to make sure Youtube videos I've watched are archived on Wayback but I have no choice but to do it manually for every link that hasn't been saved before, but with a whitelist option I could just include youtube.com in the whitelist and check the "Auto Save Page & if not archived: previously" option and be done with it. This is currently impossible to automate unfortunately.