[feature]: Back up lists to the Internet Archive

T145 / black-mirror

Blacklists and whitelists built by open code, so you know what goes into them.

GNU Affero General Public License v3.0

193 stars 12 forks source link

[feature]: Back up lists to the Internet Archive #131

Open T145 opened 7 months ago

T145 commented 7 months ago

Contact Details

No response

What's your idea?

Instead of archiving lists to a local submodule, use the Internet Archive's API and libraries to backup lists and fall back to that URL if the original can't be accessed.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

T145 commented 6 months ago

Prepend list URLs with https://web.archive.org/save/. Submitted 200 response means it gets saved successfully.

429: Save Page limits 15 requests per minute and will rate limit otherwise. This project should be fine. 509: Active session limit reached. Again this project should be fine.

T145 commented 5 months ago

Could detach the process so building can run alongside archival.

https://unix.stackexchange.com/questions/269805/how-can-i-detach-a-process-from-a-bash-script

The best plan is likely to perform archival before downloading the lists, catch any download errors, and retry those downloads with IA links.

T145 commented 3 months ago

(cb00c3f)