search

refuel-ai / autolabel

Label, clean and enrich text datasets with LLMs.
https://docs.refuel.ai/
MIT License
2.01k stars 137 forks source link

[Feature Request]: Reranking/relevance labeling task support #465

Open nihit opened 1 year ago

nihit commented 1 year ago

Is your feature request related to a problem? Please describe.

Search usecases can broadly be classified into two buckets:

  1. Lexical/Keyword based search - traditional IR systems, more or less what Lucene/Solr/ElasticSearch offer.
  2. Semantic search - embedding the doc and query in some latent space, and using a vector DB for retrieving nearest neighbors for a query.

A hybrid approach is optimal for many usecases however - (i) stage 1 - lexical similarity based retrieval & L1 ranking (ii) LLM/semantic similarity based reranking (e.g. https://txt.cohere.com/rerank/).

Describe the solution you'd like Support for Reranking task in Autolabel. Given a Query, List of Documents, and guidelines to judge relevance, ask the LLM to rank the list of documents.

aimetrics commented 8 months ago

It would be more cool if autolabel supports prompting the LLM to rank the list of images!