A clear and concise description of what you want to happen
Open Library relies on Archive.org as a storage layer for millions of book covers and has a resolver at covers.openlibrary.org which looks in the Open Library cover DB for a cover and determines where it lives (either on OL disk or in an archive.org item).
Right now w're seeing what appears to be high volume, programatic access of our covers... Instead of downloading the entire tar, they are downloading a cover at a time rapidly. This is causing performance issues on Archive.org nodes because accessing tars is much slower than zip.
We should block this high volume access first of all as a stop gap, but either way we should prioritize moving these tars to zips.
[ ] keep all tars in their archive.org cover items
[ ] produces corresponding zips for each batch
[ ] update the OL cover database to point to the zip files
[ ] test that the covers are indeed serving the zips, not the tars
[ ] remove the tars
Phase II
There is one exceptional case, which is that there is a batch of tars on disk that we'd like to be zips. One solution (while not ideal) is to perform this same process, to:
[ ] upload the tars to an archive.org item but DON'T serve from them
[ ] run Hank's derive scripts to convert those tar to zips
[ ] delete the tars from archive.org
[ ] update the OL db to use these zips
[ ] confirm they're working
[ ] remove these final remaining tars from OL disk.
Proposal & Constraints
What is the proposed solution / implementation?
Is there a precedent of this approach succeeding elsewhere?
Which suggestions or requirements should be considered for how feature needs to appear or be implemented?
Leads
Related files
Stakeholders
Instructions for Contributors
Please run these commands to ensure your repository is up to date beforecreating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
hbromley
commented
2 months ago 
cdrini
Problem
A clear and concise description of what you want to happen
Open Library relies on Archive.org as a storage layer for millions of book covers and has a resolver at covers.openlibrary.org which looks in the Open Library cover DB for a cover and determines where it lives (either on OL disk or in an archive.org item).
Right now w're seeing what appears to be high volume, programatic access of our covers... Instead of downloading the entire tar, they are downloading a cover at a time rapidly. This is causing performance issues on Archive.org nodes because accessing tars is much slower than zip.
We should block this high volume access first of all as a stop gap, but either way we should prioritize moving these tars to zips.
Prerequisites
Phase I
The strategy is:
Phase II
There is one exceptional case, which is that there is a batch of tars on disk that we'd like to be zips. One solution (while not ideal) is to perform this same process, to:
Proposal & Constraints
What is the proposed solution / implementation?
Is there a precedent of this approach succeeding elsewhere?
Which suggestions or requirements should be considered for how feature needs to appear or be implemented?
Leads
Related files
Stakeholders
Instructions for Contributors