search

openzim / zimit

Make a ZIM file from any Web site and surf offline!
GNU General Public License v3.0
331 stars 24 forks source link

Use browser-generated text as IndexData #194

Open rgaudin opened 1 year ago

rgaudin commented 1 year ago

WACZ includes a pages.jsonl file that contains a text property for every page (~HTML entries) that is extracted from the fully rendered DOM.

Using this as source for getIndexData() can be huge boost in quality for dynamic websites (building DOM in JS) versus the current situation in which the text is extracted solely from the HTML source code.

This is controlled by the --text option of the crawler.

From: https://github.com/openzim/warc2zim/issues/81

benoit74 commented 3 months ago

Is this really mandatory for 2.0 ?

benoit74 commented 3 months ago

We must still keep a fallback to indexing HTML source code, since we cannot expect pages.jsonl to be always available (warc2zim must work from only a warc file, pages.jsonl is only available when warc2zim is using in conjunction with browsertrix crawler e.g. in zimit scraper)

rgaudin commented 3 months ago

I believe this is transparent: if you have index data in pages.jsonl, then you set the getIndexData() and if you don't it's not there and libzim will index as it currently does.