Add Discord assets to global ignoreset

ArchiveTeam / ArchiveBot

ArchiveBot, an IRC bot for archiving websites

http://www.archiveteam.org/index.php?title=ArchiveBot

MIT License

356 stars 72 forks source link

Add Discord assets to global ignoreset #518

Closed Sanqui closed 1 year ago

Sanqui commented 3 years ago

^https://discord\.com/assets/

100s of urls that waste time and resources because they probably get captured on cursory crawls anyway.

tomodachi94 commented 1 year ago

@Sanqui is this on websites with the Discord widget? Just curious.

tomodachi94 commented 1 year ago

Also, looks like this was fixed in 63118fa1d65f11390fcb1d0ab9a40e221566f3c9

Sanqui commented 1 year ago

It was a problem also on websites with just a link to a Discord invite (when ArchiveBot was run with offsite links, which is the default). But yeah I got my way in 63118fa1d65f11390fcb1d0ab9a40e221566f3c9.