Add a boost to recently updated docs?

[maccman]

This is more of a discussion than an issue, but I'm curious if you have any pointers as to how we would boost documents that have been recently edited. Ideally I I would have some kind of decay function based on a date.

[lucaong]

Hi @maccman , this is possible by making use of the boostDocument search option. Here is roughly how I would implement it:

1. Add a field updatedAt to your documents, represented as an integer timestamp. Add such field to the storeFields list.
2. When searching, specify the boostDocument option as a function that calculates a boost based on updatedAt. For example, you could use an exponentially decaying function like 1 + Math.exp(-k * timeElapsedSinceUpdate), where you would choose the right value of k depending on your needs. This function will go from 2 when no time elapsed since the last update, to 1 when a long time elapsed, effectively boosting recent documents. Other functions would work too, even a simple "if more recent than 1 day than boost = 1.5, otherwise boost = 1".

Example, assuming that you want to boost documents updated more recently than 1 day ago, with a decaying boosting factor:

const miniSearch = new MiniSearch({
  fields: ['title'],
  storeFields: ['updatedAt']
})

const documents = [
  { id: 1, title: 'Some newer stuff', updatedAt: 1688984149134 },
  { id: 2, title: 'Some older stuff', updatedAt: 1688984074212 },
  # etc.
]

miniSearch.addAll(documents)

miniSearch.search('stuff', {
  boostDocument: (_id, _term, storedFields) => {
    const now = (new Date()).valueOf()
    const daysElapsed = (now - storedFields.updatedAt) / (1000 * 60 * 60 * 24)

    // Significantly boost documents more recent than 1 day, then taper off
    return 1 + Math.exp(-3 * daysElapsed)
  }
})

A few notes:

• Make sure in your calculations that the timestamps use the same format. In the example above, they are expressed in milliseconds since epoch.
• You specifically asked for a decay function, hence my example, but often it is simpler and more readable to use a function such as (daysElapsed < 1) ? 1.5 : 1. It's really up to you, just remember that a boost of 1 is the "neutral" boost (a boost of 0 would completely remove the document from the results).
• In order to use the stored fields as the third argument of the boostDocument callback, like I did in the example, you need to use at least version v6.1.0 of MiniSearch (the latest at the time of writing). For a discussion on how to do the same on older versions of MiniSearch, look at this issue.

[lucaong]

@maccman I'll go ahead and close the issue, as I think your question is solved. Feel free to continue the discussion if necessary.