Minimum should match

[subsetpark]

It would be really useful to specify "minimum should match" when using combineWith: OR. That is:

I don't want to AND every term together, but if I have 3 terms to match against, I might want 2 out of the 3 to match in order to for the document to qualify.

Right now, I'm solving that with a hack: I'm ORing together a bunch of ANDs, which are a combination of every possible minimum should match out of all the terms.

This work, but of course generating combinations can be extremely slow. I would rather that minisearch just optionally counted the number of terms that matched and only returned when it hit the minimum.

Thank you for the great library!

[lucaong]

Hi @subsetpark , yours is an interesting idea. One reason why I am not sure whether to implement it as a feature though, is that there is already an easy way to implement this on top of MiniSearch, which does not rely on generating all combinations. You could perform a standard search, and use the filter option to only retain results where the terms array is longer than your "minimum should match":

const miniSearch = new MiniSearch({
  fields: ['text']
})

const documents = [
  { id: 1, text: 'Something ordinary' },
  { id: 2, text: 'Something extraordinary' },
  { id: 3, text: 'Something truly extraordinary' },
  { id: 4, text: 'Something somehow strange' }
]

miniSearch.addAll(documents)

miniSearch.search("something truly extraordinary", {
  combineWith: 'OR',
  filter: (result) => result.terms.length >= 2
})
/* =>
[
  {
    id: 3,
    score: 8.584958632398532,
    terms: [ 'something', 'truly', 'extraordinary' ],
    match: { something: [Array], truly: [Array], extraordinary: [Array] }
  },
  {
    id: 2,
    score: 2.5275597943271197,
    terms: [ 'something', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  }
]
*/

As you can see, even though we are combining with OR (and therefore all four documents in the example should match at least one term), only the results that match at least 2 terms are returned.

There is one situation with "minimum should match" that can be confusing, which occurs when using fuzzy or prefix match: the terms array reflects the matched terms in the result documents, not in the search query. Therefore, one might meet the "minimum should match" even if fewer query terms match, when a single query term expands to multiple matches:

// Using the same data as above

miniSearch.search("some extraordinary", {
  combineWith: 'OR',
  prefix: true,
  filter: (result) => result.terms.length >= 2
})
/* =>
[
  {
    id: 2,
    score: 2.3012539398292353,
    terms: [ 'something', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  },
  {
    id: 3,
    score: 2.0778819231808248,
    terms: [ 'something', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  },
  {
    id: 4,
    score: 0.6200953275356065,
    terms: [ 'somehow', 'something' ],
    match: { somehow: [Array], something: [Array] }
  }
]
*/

In the results above, one might expect the document with ID 4 to be excluded, as it only matches one query term, but it is included as some also expands to something due to prefix search. It's really up to the specific application how this case should be treated, therefore a simple "minimum should match" option is not a one-size-fits-all solution.

A feature that I think might help, would be to add a queryTerms array to the MiniSearch results (that in the case above would only contain some for the result with ID 4): this way, one could define custom logic based on either the number of query terms matched, or the number terms matched in the results.

[subsetpark]

A feature that I think might help, would be to add a queryTerms array to the MiniSearch results (that in the case above would only contain some for the result with ID 4): this way, one could define custom logic based on either the number of query terms matched, or the number terms matched in the results.

@lucaong thanks for your insights here; I hadn't thought about the terms list in filter and will see if that satisfies my needs. Per your comment here, it might not; I might really care about minimum terms in the query, not in the document. But you're right that it's not going to be the same for everyone.

I like the idea mentioned here as well because I was yesterday wishing that I also had access, in the result, to how the terms match. That is: I might want to filter out a result if all the matches are fuzzy, or by prefix, rather than whole term matches.

[lucaong]

Here's a pull request introducing queryTerms to search results.

[lucaong]

And it's now released on NPM as part of version v6.3.0. I would therefore consider this issue closed, but @subsetpark feel free to comment further if necessary.

For reference, here is the revised solution, using the new queryTerms result field:

const miniSearch = new MiniSearch({
  fields: ['text']
})

const documents = [
  { id: 1, text: 'Something ordinary' },
  { id: 2, text: 'Something extraordinary' },
  { id: 3, text: 'Something truly extraordinary' },
  { id: 4, text: 'Something somehow strange' }
]

miniSearch.addAll(documents)

miniSearch.search("something truly extraordinary", {
  combineWith: 'OR',
  filter: (result) => result.queryTerms.length >= 2
})
/* =>
[
  {
    id: 3,
    score: 8.584958632398532,
    terms: [ 'something', 'truly', 'extraordinary' ],
    queryTerms: [ 'something', 'truly', 'extraordinary' ],
    match: { something: [Array], truly: [Array], extraordinary: [Array] }
  },
  {
    id: 2,
    score: 2.5275597943271197,
    terms: [ 'something', 'extraordinary' ],
    queryTerms: [ 'something', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  }
]
*/

// And now also the example using prefix or fuzzy match works as expected:

miniSearch.search("some extraordinary", {
  combineWith: 'OR',
  prefix: true,
  filter: (result) => result.queryTerms.length >= 2
})
/* =>
[
  {
    id: 2,
    score: 2.3012539398292353,
    terms: [ 'something', 'extraordinary' ],
    queryTerms: [ 'some', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  },
  {
    id: 3,
    score: 2.0778819231808248,
    terms: [ 'something', 'extraordinary' ],
    queryTerms: [ 'some', 'extraordinary' ],
    match: { something: [Array], extraordinary: [Array] }
  }
]
*/

Note that by intersecting terms and queryTerms you can also find which terms matched exactly.

I hope this solves your case :) Thanks again for opening this issue, I think this is a useful addition to MiniSearch.