Consider using PouchDB to replace custom storage/search solution

openzim / gutenberg

Scraper for downloading the entire ebooks repository of project Gutenberg

https://download.kiwix.org/zim/gutenberg

GNU General Public License v3.0

126 stars 37 forks source link

Consider using PouchDB to replace custom storage/search solution #91

Closed kelson42 closed 4 years ago

kelson42 commented 4 years ago

The problem with current solution is that it will be difficult to implement additional search/filter capabilities. If we do so with what we have the problematic of crossing result will be serious. We might instead use PouchDB is a full offline approach https://pouchdb.com/

kelson42 commented 4 years ago

@darkenvy has a working POC I believe :)

psiie commented 4 years ago

poc2.zip

Im seeing 22s import time on desktop. 45s import time on Pixel2 phone. After that, searching is indexed in the background (searching can occur before indexing is done :) ) and once indexed see instant results on the desktop, and 2s result time on mobile.

How to run the POC

extract to directory
run python -m SimpleHTTPServer
preferably open developer console in browser before navigating to localhost:8000
open localhost:8000. See the database load in the console.

kelson42 commented 4 years ago

I think @darkenvy implemented this ticket. We should now figure out about the software architecture of the second generation scraper. A topic which should not be discussed here.