Indexing with AWS-S3 Bucket is failing

[ApsaraDhanasekar11]

Hi, I tried to create index with a different backend (AWS -S3 bucket) using the s3leveldown module as a DB store option. The Index is being created, but while querying using _SEARCH/ QUERY methods, the result set is inappropriate. Like for eg, when I initialise the DB with the S3 bucket and use the PUT method to add documents, where my text is "Final is the file name".. and "what is the version" . This is how its created ::

current Indexed one:: { key: 'description:file#0.60', value: [ '1635744247556-1-1' ] }. Another one:: { key: 'description:version#0.50', value: [ '1635744285856-1-1' ] }

I am able to see the above in my store, when I do a createReadStream. But when my search keyword is "version", my expected result should be only the 2nd indexed document. But it gives me both 1st and 2nd.I tried using both _SEARCH/ QUERY methods, but both r giving same wrong/additional results. I took reference from the below test folder examples:: https://github.com/fergiemcdowall/search-index/blob/master/test/src/memdown-test.js . Can someone guide on the correct approach for implementing other backend store options like Amazon-S3 ?

[fergiemcdowall]

Thanks for the bug report @ApsaraDhanasekar11. I think I understand your problem, but there are too many variables to reproduce it accurately.

Could you include a standalone script/test that demonstrates the issue?

[ApsaraDhanasekar11]

Hi @fergiemcdowall , thanks for replying back. Please find the below example code and help us with details. Thanks!

const levelup = require('levelup'); const si = require('search-index'); const s3leveldown = require('s3leveldown');

const s3Store = await levelup(s3leveldown(bucketName, S3Client));

const idx = await si({ db: s3Store, storeVectors: true });

let data = [ { _id: 'a', description: 'Use template to list' }, { _id: 'b', description: 'All versions and updates' }, { _id: 'c', description: 'Final is the file name' } ];

const result = await idx.PUT(data, { storeVectors: true });

// results is :: [ { _id: 'a', operation: 'PUT', status: 'CREATED' }, { _id: 'b', operation: 'PUT', status: 'CREATED' }, { _id: 'c', operation: 'PUT', status: 'CREATED' } ]

// ** The above code creates the index as below::
{ key: 'description:file#1.00', value: [ 'c' ] } { key: 'description:final#1.00', value: [ 'c' ] } { key: 'description:list#1.00', value: [ 'a' ] } { key: 'description:name#1.00', value: [ 'c' ] } { key: 'description:template#1.00', value: [ 'a' ] } { key: 'description:updates#1.00', value: [ 'b' ] } { key: 'description:use#1.00', value: [ 'a' ] } { key: 'description:versions#1.00', value: [ 'b' ] } { key: '￮DOCUMENT_COUNT￮', value: 3 } { key: '￮DOC_RAW￮a￮', value: { _id: 'a', description: 'Use template to list' } } { key: '￮DOC_RAW￮b￮', value: { _id: 'b', description: 'All versions and updates' } } { key: '￮DOC_RAW￮c￮', value: { _id: 'c', description: '"Final is the file name' } } { key: '￮DOC￮a￮', value: { _id: 'a', description: [ 'list#1.00', 'template#1.00', 'to#1.00', 'use#1.00' ] } } { key: '￮DOC￮b￮', value: { _id: 'b', description: [ 'all#1.00', 'and#1.00', 'updates#1.00', 'versions#1.00' ] } } { key: '￮DOC￮c￮', value: { _id: 'c', description: [ 'file#1.00', 'final#1.00', 'is#1.00', 'name#1.00', 'the#1.00' ] } } { key: '￮FIELD￮description￮', value: 'description' } //// ***** ///////

// ** For the Search/ query : *** // const result = await indexedDb.QUERY( { GET: { FIELD: ['description'], VALUE: { GTE: 'versions', LTE: 'versions' }, } }); ----> Tried other options like Query->(GET, SEARCH) , _SEARCH, _GET

But the result was :: RESULT: [ { _id: 'c', _match: [ 'description:file#1.00', 'description:final#1.00', 'description:name#1.00' ] }, { _id: 'a', _match: [ 'description:list#1.00', 'description:template#1.00', 'description:use#1.00' ] }, { _id: 'b', _match: [ 'description:updates#1.00', 'description:versions#1.00' ] } ], RESULT_LENGTH: 3 } :: which is actually giving results of all previous alphabetical words from a-v (as "versions" begins with v)

While trying to identify the process flow, I noticed the GET function has internal implementation of db.createReadStream method which should actually filter the data according to the keywords passed in GTE & LTE. But looks like this is failing and instead bringing up the entire result set restricting upto the first character (alphabetic order)..