Palm Leaf crawl - Githubissues

internetarchive / dweb-mirror

Offline Internet Archive project

https://www-dweb-mirror.dev.archive.org/

GNU Affero General Public License v3.0

273 stars 31 forks source link

Palm Leaf crawl #304

Closed mitra42 closed 4 years ago

mitra42 commented 4 years ago

Need to be able to crawl the palmleaf

[ ] Run test query 5 items (example below)
- [x] Seeing Forbidden - was epub's - handled
- [x] Check getting all pages
- [x] Check getting right sizes - probably not - want full size + 2000px + 400px (see below)
- [x] Check rough size ~2.26Mb per page, ~ 100Mb (tested on 1 page only); ... target 10GB
[x] Need version of SD on bigger card - determine size
[x] Copy over to SD
[ ] Check on RPI

mitra42 commented 4 years ago

Test query as currently in yaml

      - query: 'collection:Bali AND external-identifier:*palmleaf.org*'
        level: details
        search:
          sort: '-downloads'
          rows: 5
          level: details

mitra42 commented 4 years ago

Getting right size

[x] CrawlManager.CrawlItem._crawlPages pass idealWidth to pageParams
[x] Detect palmleaf and then fetch 3 pages [400,2k] and scale =1

mitra42 commented 4 years ago

Works as saved search and now as collection bali-lontar-transcribed