Create Semantic Ranking for Title, Summary, Keywords, Title and Conference Name Searching

[nbcstevenchen]

Implemented 2 methods for semantic ranking.

BM25 algorithm

• A widely used ranking algorithm based on term frequency and inverse document frequency, without relying on machine learning.
• Fast
• Not bad accuracy .

Sentence BERT Bi-Encoder

• Deep learning based method, which do the semantic search based on the semantic information.
• Slower than BM25, but achieves reasonable speed performance on CPU inference.
• Good accuracy.

I have done some tests on the 600 samples that we extracted. Both methods works well.

[rootfs]

@nbcstevenchen this is very cool! Do you need GPU to run the similarity search? Can it be done on github runner?

[nbcstevenchen]

@nbcstevenchen this is very cool! Do you need GPU to run the similarity search? Can it be done on github runner?

I don't think GPU is necessary. The Bi-Encoder is specifically designed to facilitate faster inference speeds in applications by enabling the pre-generation of embeddings. BM25 also works well. We can use this one first for the demo.