wggb / wiki-website

0 stars 0 forks source link

Document Search #4

Open aeirya opened 6 months ago

aeirya commented 6 months ago

We need a fast way to find related documents to the given user query. I suggest a combination of:

TF-IDF vector space or the enhanced BM25 to improve speed
Sentence BERT to embed sentences of each document and compare sentences faster
Normal BERT or ColBERT for best semantic comparison between two sentences

A TF-IDF + SBERT + BERT combination will most likely work well.

aeirya commented 6 months ago

3 Covers this to some extent. More development is needed.