Counterbalance popularity

[roopeux]

Create a new field in mappings called freshness.
Get the first date of popularity data collection and give all docs published before this date the min value of pop_score, which is currently 0.4. Give docs published today value 1. For the docs published during the pop data collection, calculate freshness based on the number of days on a linear function between min pop_score and 1, later days creating higher values.

Explanation This makes new publications as popular as the most popular old texts for a while, giving them a chance to gain actual popularity.

[eroux]

I understand the use case but from an engineering perspective this is something I'd rather avoid having hard coded in the documents, because it's a field where the value changes all the time, so we would need to reindex everything every day for this field to work. Is there a way this could be calculated a query time? there's already a publicationDate field that indicates when the record was created.

Note that even if we implement that knowing it's going to be very imperfect, there's still different dates for a bibliographical record:

• record creation date (recorded in the field "publicationDate")
• last record modification date
• first time the record had images online (this can happen a few months / years after the creation of the record), currently recorded in the field "firstScanSyncDate"
• last time images were modified (images added or improved)
• first time the record had an etext online (I guess I should record that, even though it's a bit more difficult to access for now)
• last time an etext was modified

[eroux]

there's also the case of non-bibliographical records: I tend to think that the records that we're adding today are not that relevant compared to old records

[eroux]

what about having 2026-01-01 - date / 2026-01-01 - 2007-01-01 as the score? that way we can just update it in 2026

[roopeux]

I can calculate this at query time as you suggested. My idea was that it was not necessary to reindex every day because it does not have to be accurate, and while some texts are the newest, even if they were not published today, it would be ok to keep boosting them. But since latencies are in control, you don't have to do it.

[eroux]

added a scans_freshness field which is the date put in the interval [2016-05-01 - 2026-05-01], I think it should be doable to adjust it at query time so it boosts consistently over time