search

pouchdb / pouchdb-server

CouchDB-compatible server built on PouchDB and Node
Apache License 2.0
951 stars 154 forks source link

Very slow indexing #314

Open carrotalan opened 6 years ago

carrotalan commented 6 years ago

We currently have a project that uses express-pouchdb with 12k small documents (about 20 lines per JSON document) in one database. It uses the default leveldown as its db backend and 15 indexes all within the same design document.

When ever we make a PUT/POST of more than 100 documents (all done via single requests), it can take 5-10 minutes to reindex all the views again.

It is running from a modern Intel NUC with an i5 CPU, 4GB RAM and Windows 7 32 bit. Is this considered normal? What could we consider to speed up the indexing process.

mikeymckay commented 5 years ago

If you use 15 indexes in one design document, then any change to any one of those indexes will require all 15 indexes to be rebuilt. This means that 12,000 documents are opened 15 times and each index executed against that document. Even your 100 documents PUT/POST will require the index to be run 1500 times.

Better to have one index per design document. Also, consider carefully the complexity of your map/reduce code. This post is the best I've read on indexing strategies:

https://pouchdb.com/2014/05/01/secondary-indexes-have-landed-in-pouchdb.html