Recently we introduced a feature that will let the Dry Archive's directory structure be used behind Refresh's normal data store. This means that all hashes stored in the archive can be referenced in addition to Refresh's users' content.
Using our API is much much faster than downloading from Archive.org, since we've downloaded the full archive on our server and extracted it so everything's right there on the disk, so no parsing the zip files, no reaching out to archive.org, etc.. In addition, we have CloudFlare caching all the assets once downloaded.
I tested this with one of my levels and downloading just 7 assets that are about 220K in size is much much faster, beating archive.org by a few seconds.
This PR uses that public API to speed things up, but uses archive.org as a fallback for when things go south on our end.
As a side-note, I'm not too familiar with Rust so apologies if I'm breaking any conventions, but what I did seems to work pretty well.
Recently we introduced a feature that will let the Dry Archive's directory structure be used behind Refresh's normal data store. This means that all hashes stored in the archive can be referenced in addition to Refresh's users' content.
Using our API is much much faster than downloading from Archive.org, since we've downloaded the full archive on our server and extracted it so everything's right there on the disk, so no parsing the zip files, no reaching out to archive.org, etc.. In addition, we have CloudFlare caching all the assets once downloaded.
I tested this with one of my levels and downloading just 7 assets that are about 220K in size is much much faster, beating archive.org by a few seconds.
This PR uses that public API to speed things up, but uses archive.org as a fallback for when things go south on our end.
As a side-note, I'm not too familiar with Rust so apologies if I'm breaking any conventions, but what I did seems to work pretty well.