Open giuliac89 opened 6 years ago
How slow is the indexing performance? How many documents are you indexing? Its difficult to know for sure without know what the addXMetadata
plugins are doing.
How much metadata are you storing against each term? Storing a lot of metadata can make the index very large.
I index about 1400 documents and for each term I store 8 metadata. The plugin implementation is simple:
function addXmlDocIdMetadata(builder, parsedElementsForIndexing) {
var pipelineFunction = function (token) {
var docIndex = builder.documentCount - 1;
token.metadata['xmlDocId'] = parsedElementsForIndexing[Object.keys(parsedElementsForIndexing)[docIndex]].xmlDocId;
return token;
};
lunr.Pipeline.registerFunction(pipelineFunction, 'xmlDocId');
builder.pipeline.add(pipelineFunction);
builder.metadataWhitelist.push('xmlDocId');}
The index performance is around 30 seconds...
Does the 30 seconds include serialising the index to JSON, e.g. are you doing this:
JSON.stringify(idx)
If you can share the data you are indexing, as well as the code to do the indexing, I can take a look.
It doesn't include serialising. You can find the project I'm working on at https://github.com/evt-project/evt-viewer and my branch is feature/search . After the environment setup you have to create a data folder under app folder, where you have to add a xml file. My code is under app/src/dataHandler/search/searchIndex.service.js You can download the xml file here: https://github.com/arojascastro/soledades/blob/master/edicion/anotada/input/soledades_anotada.xml
Follow the readme file for more information. Thanks!
Hi Oliver,
indexing performance is slow when I index lots of documents with metadata. An example of document to be indexed is this
And I build the index like that
Is there a way to improve performance?