search

Lyken17 / hf-torrent

34 stars 1 forks source link

[Enhancement] Auto-Upload to Internet Archive #1

Open ZaxLofful opened 6 months ago

ZaxLofful commented 6 months ago

A buddy and me were looking to make this exact thing, except we were wanting to take it one step farther.

After the torrent is created, automatically upload it and create a page for the model on the Internet Archive.

The Internet Archive, already crawls and stores all the web pages for every hugging face model, but it doesn't capture the binary files or weights.

Storing this on the Internet Archive, with a like to the already crawled and stored Hugging Face pages; will be a complete and permanent archive of the info.

Lyken17 commented 6 months ago

Sounds cool! The genreated torrent files are already stored at https://github.com/lyken17/hf-torrent-store/. You are welcome to store them on the internet archieve

ZaxLofful commented 6 months ago

I meant as part of the script, it would generate a page for each torrent and then link them to the IA page for them. You are welcome to mirror your own stuff on IA, if all you want to do is store your own.

Lyken17 commented 6 months ago

I am not familiar with IA and do have the bandwidth to support the feature now.

If someone would like to contribute I can merge the PR