search

webrecorder / browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container
https://crawler.docs.browsertrix.com
GNU Affero General Public License v3.0
613 stars 79 forks source link

Investigate automatic blocking of cookie popups #182

Closed ikreymer closed 8 months ago

ikreymer commented 1 year ago

Brave recently added a feature for blocking cookie popups, I believe it is using: https://easylist.to/ cookie list. If this works well enough as a Brave feature for users, we should definitely look into adding this as an option for the crawler! I assume this could work similarly to the ad block, where we download the list and then apply it as part of the request handling to add cookies?

tw4l commented 1 year ago

The question is whether to implement in the browsertrix-crawler code via #187 or to switch to using Brave in the crawler with a custom profile, which would net the same result and possibly have other benefits as well.

tw4l commented 1 year ago

Related to https://github.com/webrecorder/browsertrix-crawler/issues/189

tw4l commented 8 months ago

Now that we're using Brave, this is doable via a browser profile using Brave's build-in features, so closing!