search

Hydriz / Balchivist

Python library for archiving datasets
GNU General Public License v3.0
5 stars 1 forks source link

Archive shorturls dumps #8

Open legoktm opened 4 years ago

legoktm commented 4 years ago

Hi, would it be possible to back up the shorturls dumps (https://dumps.wikimedia.org/other/shorturls/) to archive.org? @nemobis pointed me in your direction.

Hydriz commented 4 years ago

Yep it's definitely possible. However, I am in the midst of rewriting the backend to be more robust so it will take some time before the files appear on Archive.org, along with the other datasets that Wikimedia provides.

legoktm commented 4 years ago

Awesome. How long do you think that will take? We want to start cleaning up the old shorturls dumps (see https://phabricator.wikimedia.org/T257782). If it's going to be a while, I can back them up to IA manually for now until your system is ready.