search

refuel-ai / autolabel

Label, clean and enrich text datasets with LLMs.
https://docs.refuel.ai/
MIT License
2.1k stars 149 forks source link

[Feature Request]: Reranking/relevance labeling task support #465

Open nihit opened 1 year ago

nihit commented 1 year ago

Is your feature request related to a problem? Please describe.

Search usecases can broadly be classified into two buckets:

  1. Lexical/Keyword based search - traditional IR systems, more or less what Lucene/Solr/ElasticSearch offer.
  2. Semantic search - embedding the doc and query in some latent space, and using a vector DB for retrieving nearest neighbors for a query.

A hybrid approach is optimal for many usecases however - (i) stage 1 - lexical similarity based retrieval & L1 ranking (ii) LLM/semantic similarity based reranking (e.g. https://txt.cohere.com/rerank/).

Describe the solution you'd like Support for Reranking task in Autolabel. Given a Query, List of Documents, and guidelines to judge relevance, ask the LLM to rank the list of documents.

aimetrics commented 10 months ago

It would be more cool if autolabel supports prompting the LLM to rank the list of images!