Added covers appear broken

[seabelis]

Problem

I've noticed some covers appear broken even though a valid cover has been uploaded. Example, https://openlibrary.org/books/OL51711917M/The_best_ghost_stories This does not happen consistently, so I cannot provide steps to reproduce. This cover was added in May 2024, but now appears broken. The edit cover modal indicates no cover was uploaded. Where did it go? There's no history of the cover being removed.

Reproducing the bug

1. Go to ...
2. Do ...

• Expected behavior:
• Actual behavior:

Context

• Browser (Chrome, Safari, Firefox, etc):
• OS (Windows, Mac, etc):
• Logged in (Y/N):
• Environment (prod, dev, local): prod

Breakdown

Requirements Checklist

• [ ]

Related files

*

Stakeholders

*

---

Instructions for Contributors

• Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[seabelis]

Here's another example. https://openlibrary.org/works/OL3360282W/Classic_American_Short_Stories?m=history

[scottbarnes]

Some additional information: for the first book, the cover, 14627720-L.jpg, seems to be missing from https://ia600505.us.archive.org/view_archive.php?archive=/35/items/l_covers_0014/l_covers_0014_62.zip.

[mekarpeles]

It looks like there was an error very specifically with https://ia600505.us.archive.org/view_archive.php?archive=/23/items/covers_0014/covers_0014_62.zip

i.e. covers_0014 (all sizes) being uploaded when that batch in particular was incomplete. We're investigating whether this was the cover archive pipeline or the result of a manual upload. The other batches seem unaffected but these 5k or so covers are likely gone / cleaned up by the finalize step after it noticed the batches were uploaded to archive.org.

While investigating...

• We found covers still in ol-covers0:/1/var/lib/openlibrary/coverstore/localdisk but most we checked looked like duplicates, corrupt, or authors. It's a good reminder that maybe author images are not currently being archived yet?
• Also, the localdisk directory still has a ton of folders, which maybe isn't great and should be cleaned/pruned either by our cron or manually.

Possible interventions are:

• [ ] The upload step should call an updated is_uploaded, which checks both if/that the zip is complete and if any uploaded zip is complete, e.g. matches the hash of the completed local zip
• [ ] The update should only mark covers as uploaded if we run
• [ ] The finalize step should also check the new is_uploaded function and should not delete local zips or local covers unless (a) the local zip is complete, (b) uploaded, and (c) the uploaded zip hash matches local zip hash