Working with offline content from Jake's tool for Carl

internetarchive / dweb-mirror

Offline Internet Archive project

https://www-dweb-mirror.dev.archive.org/

GNU Affero General Public License v3.0

261 stars 27 forks source link

Working with offline content from Jake's tool for Carl #280

Open mitra42 opened 4 years ago

mitra42 commented 4 years ago

See #64 (compatability with Jake's tool)

Carl Malamud has a use case where he has downloaded, using Jake's tool, a bunch of items for storage and availability offline in India.

This item is a Meta task for all the challenges of working with it :-)

See the #carl branch of dweb-mirror for any code below, its not merged because its currently incomplete, mostly untested and adds about 10Mb of dependencies without currently adding functionality.

[ ] Phase 1: download missing files
[ ] Phase 2: Reading xml and zips instead of json

mitra42 commented 4 years ago

[x] Reading xml instead of json

Problem is that Jake's tool downloads files like _meta.xml rather than the JSON.

Added shim to ArchiveItemPatched.read -> _parse_xml

mitra42 commented 4 years ago

[ ] There is no equivalent of the _extras.json Specificially the "server" and "dir" are required for books, if the book server is to be used

mitra42 commented 4 years ago

[ ] Books -
- If use server need the "server" and "dir" in the _extras.json
- better would be to use the Zip directly, but then it needs:
  - zip extraction
  - image scaling (though a locally extracted zip might also be ok)
- [ ] Need an example in the downloaded files.
[ ] Note that the bookreader manifest data isn't stored in the item in any usable manner :-(

Fallback If have internet then can re-download metadata/json then get correctly scaled pages