Closed PJ-Finlay closed 3 years ago
Related to this once better distribution is in place we'll want to add the ability to manage model downloads from the GUI.
GitHub has (so far, crossing fingers) been pretty good at hosting binary data (up to ~2GB in size per file) without limits, although the party might end sometimes in the future.
Otherwise a central server with CDN works pretty well too. For cheap, there's https://www.digitalocean.com/products/spaces/ (although careful on the outbound transfer fees!) or https://wasabi.com/ (no egress fees)
A CDN is probably the best long term solution (in addition to providing torrents) and Google Drive is definitely not ideal. Last weekend when there was a lot of traffic from Hacker News it seems that some people weren't able to download models. I'm tempted to link directly to the githubusercontent url in the LibreTranslate Models repo but that may increase the chance that we can't continue to use GitHub.
https://stackoverflow.com/questions/38768454/repository-size-limits-for-github-com
I'm not too worried; the discussion on S.O. is pre-Microsoft acquisition. If you want to link it, go ahead.
I wonder if the raw data is also stored anywhere. Or do we only have the compiled model files?
No, the training data is very large. I don't even store it myself I just re-download from Opus every time I train a new model. If you're interested in the data go to Opus. There's also a generate_wiktionary_data.py
script in the training scripts.
We're now sharing one model pair (English-Chinese) via IPFS (available on package index) as a test. A volunteer is currently pinning for us and we can expand if people find this useful.
It might be useful to consider using GitHub Releases to distribute the models, as there are no bandwidth limitations and since CloudFront is used to deliver the assets.
Currently models are distributed by Google Drive (not ideal) and a slow BitTorrent, so there's lots of room for improvement:
The plan was to make a separate repo for storing model distribution information so let me know if your interested.