NEOS-AI / Neosearch

AI-based search engine done right
Apache License 2.0
7 stars 0 forks source link

Hybrid Search (Dense Vector Search + Sparse Vector Search) #87

Open YeonwooSung opened 4 months ago

YeonwooSung commented 4 months ago

llama-index: qdrant hybrid search

dense vectors are the ones you have probably already been using -- embedding models from OpenAI, BGE, SentenceTransformers, etc. are typically dense embedding models. They create a numerical representation of a piece of text, represented as a long list of numbers. These dense vectors can capture rich semantics across the entire piece of text.

sparse vectors are slightly different. They use a specialized approach or model (TF-IDF, BM25, SPLADE, etc.) for generating vectors. These vectors are typically mostly zeros, making them sparse vectors. These sparse vectors are great at capturing specific keywords and similar small details.