jkomoros / card-web

The web app behind thecompendium.cards
Apache License 2.0
56 stars 8 forks source link

Better way to store expensive indexes #315

Open jkomoros opened 4 years ago

jkomoros commented 4 years ago

Originally captured in #151.

There are a few types of indexes that are expensive to compute, inncluding tfidf for cards, and card_rank.

These need to be updated every time that a card is updated, but aren't THAT expensive. tfidf can change more slowly than card_rank may.

For both of these ideally every client doens't have to constantly recompute.

In addition, card_rank should be used for sorts, but sort functions only take in a card. Ideally there's some way to mix in data into a card like card_rank. Or I guess just make it so sorts also take card-rank.

Note that the pagerank library is 314kb at least to include. Ideally it wouldn't be included unless the user explicitly sorted by card-rank.

Another approach is to have an indexes collection, one for tfidf and one for card-rank that are fetched and kept updated. But that could be a lot of data to update, and it still would need to be merged in somewhere into the cards to be used for sorting.

jkomoros commented 4 years ago

Potentially related to #367