How to create indexes syncing with pouchdb

[AnnamrajuSarvani]

Hi, I have a local pouch database, I want to implement search on pouchdb using search-index. Is it possible? If it is possible, how to do that? What is the best possible way for that?

[fergiemcdowall]

Its certainly possible- check out slides 24-25 here: https://speakerdeck.com/fergiemcdowall/an-introduction-to-search-index-dot-js (this stuff is a couple of years old- APIs may have evolved slightly)

Indexing generally takes some time, replication is faster. For most use cases it makes sense to generate an index outside of the browser and then replicate it into the browser.

[AnnamrajuSarvani]

I am getting this error while creating: <--- Last few GCs --->

255751 ms: Scavenge 1400.7 (1457.4) -> 1400.7 (1457.4) MB, 7.8 / 0 ms (+ 0.1 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep]. 255851 ms: Mark-sweep 1400.7 (1457.4) -> 1396.2 (1457.4) MB, 99.7 / 0 ms (+ 47.7 ms in 215 steps since start of marking, biggest step 1.5 ms) [last resort gc]. 255945 ms: Mark-sweep 1396.2 (1457.4) -> 1396.2 (1457.4) MB, 94.0 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x95013e3ac1 1: replace(aka replace) [native string.js:~152] [pc=0x81e0bc5c25c] (this=0xab351caeb41 <Very long string[8056]>,P=0xab351caeaf9 ,Q=0x3e7d8e622aa9 <String[1]\: \x00>) 2: parseIndexableString(aka parseIndexableString) [/home/cronj-10/Sarvani/mongo-pouchdb-convertor/node_modules/pouchdb/lib/index.js:~3297] [pc=0x81e0bbb7a8a] (this=0x9501304189 ,str=0x6e539e73dc9 <Very l...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory Aborted (core dumped)

[fergiemcdowall]

Thats an "out of memory" error

[AnnamrajuSarvani]

What is the solutions for that?

[eklem]

Give the node-process more memory:

$ node --max-old-space-size=2048 [your node  script]

Here 2 GB is allocated

[AnnamrajuSarvani]

Let me try...

[fergiemcdowall]

@AnnamrajuSarvani just to clarify- are you using PouchDB or CouchDB?

[AnnamrajuSarvani]

PouchDB

[fergiemcdowall]

In a browser, or on a computer?

[AnnamrajuSarvani]

computer, I am working on desktop application.

[fergiemcdowall]

Ahhh- gotcha

[AnnamrajuSarvani]

I am trying to create index for 20 records. I don't get any output see lines below.

var db= new PouchDB('sample'); var options = { indexPath: 'pouchIndex', logLevel: 'error' } //put the startup options you want here var SearchIndex = require('search-index') SearchIndex(options, function(err, si) { db.allDocs({includedocs:true,limit:20}) .then(function(dataSet){ si.add({'batchName':'pouchdb'},.map(dataSet.rows,processData),function(err){ if(!err) console.log("no err"); else console.log(err) })

})
.catch(function(err){
  console.log(err);
});

}); What is the issue?

[AnnamrajuSarvani]

Creation fails for me. I tried max, It doesn't create anything. Any alternative for this?

[AnnamrajuSarvani]

I tried all possible cases mentioned in documentation.

[fergiemcdowall]

Hi @AnnamrajuSarvani - We know that reading from pouchdb to search-index is possible, but it hasnt been documented for a couple of years, and the APIs have probably changed ever so slightly. If you get it working, we would love to have a HOWTO to help others with the same use case.

[AnnamrajuSarvani]

Hi, I am creating json files to sync the data. Now while reading file and creating index I get this error:

[OpenError: IO error: lock cases_index/LOCK: already held by process]

How to close leveldb and create next index? Is there any way to close the index?

[fergiemcdowall]

yes, read the API docs for .close()

[AnnamrajuSarvani]

How to do that? fs.createReadStream('DataJson/'+filenames[i])
.pipe(index.feed())

          .on('finish', function() {
                // I want to close index here
           });

[fergiemcdowall]

.on('finish', function() {
  index.close(function(err) {
    // index is now closed
  })
});

[AnnamrajuSarvani]

Let me try...

[AnnamrajuSarvani]

It works, Thanks. Bulk create for index doesn't work for my data as it is huge. So I am creating json files in one directory. Each JSON file has 100 records. Now I am reading files from my directory and creating data to index using below code. createIndex(fileNames,i){ fs.createReadStream('DataJson/'+filenames[i]) .pipe(index.feed()) .on('finish', function() { // I want to close index here index.close(function(err){ if(!err){ createIndex(fileNames,i) } })

       });

}

I made it synchronous because index will be locked. I need to check whether it works for .forEach or async loop.

[fergiemcdowall]

This issue is becoming a bit more of an ongoing conversation, so closing.

You can also get us here: https://gitter.im/fergiemcdowall/search-index