Knowledge base setup, mixed search requires enabling the Rerank model, but only LocalAI supports the Rerank model locally. Is there a complete example?

Hello @idefa! I'm here to help you with any bugs, questions, or contributions you may have. I'll do my best to assist you while we wait for a human maintainer.

To set up a knowledge base with the Rerank model for mixed search using LocalAI, you can follow these steps:

Create a Knowledge Base Application:
- Go to Studio -- Create Application -- Create Chat Assistant.
- Enter Context Settings, click Add, and select the already created knowledge base.
- In Context Settings -- Parameter Settings, configure the Recall Strategy.
- Enable Citation and Attribution in Add Features.
- In Debug and Preview, input user questions related to the knowledge base for debugging.
- After debugging, Save and Publish as an AI knowledge base Q&A application.
Choose a Recall Mode:
- N-Choose-1 Recall: The LLM autonomously selects the most relevant single knowledge base to query for related text.
- Multi-Path Recall: Matches all knowledge bases simultaneously based on user intent, querying related text fragments from multiple knowledge bases. After a re-ranking step, the best result matching the user question is selected from the multi-path query results. This mode requires the configuration of the Rerank model API.
Configure the Rerank Model:
- Dify supports the Cohere Rerank model. Enter the "Model Provider -> Cohere" page and fill in the Rerank model's API key.
- You can also use local inference frameworks like Ollama and Xinference to deploy local Rerank models such as bge-reranker.
Set the Rerank Model:
- Go to the "Dataset -> Create Dataset -> Retrieval Settings" page and add the Rerank settings.
- Set TopK to determine the number of relevant documents returned after Rerank.
- Set Score Threshold to determine the minimum score of relevant documents returned after Rerank.
- When setting the recall mode to Multi-Path Recall in the "Prompt Orchestration -> Context -> Settings" page, enable the Rerank model.

Here is an example of how you can set up and use the LocalaiRerankModel:

import os
from core.model_runtime.entities.rerank_entities import RerankResult, RerankDocument
from core.model_runtime.model_providers.localai.rerank.rerank import LocalaiRerankModel

# Initialize the LocalaiRerankModel
rerank_model = LocalaiRerankModel()

# Define model credentials
credentials = {
    'server_url': os.environ.get('LOCALAI_SERVER_URL'),
    'api_key': 'your-api-key'
}

# Define the model name
model_name = 'bge-reranker-base'

# Define the search query and documents
query = "What is the capital of the United States?"
docs = [
    "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274.",
    "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan."
]

# Invoke the rerank model
try:
    rerank_result: RerankResult = rerank_model._invoke(
        model=model_name,
        credentials=credentials,
        query=query,
        docs=docs,
        score_threshold=0.8,
        top_n=5,
        user="abc-123"
    )
    # Process the rerank result
    for doc in rerank_result.docs:
        print(f"Document: {doc.text}, Score: {doc.score}")
except Exception as e:
    print(f"Error invoking rerank model: {e}")

This code sets up the LocalaiRerankModel, validates the credentials, and invokes the rerank model with a search query and documents. The results are then processed and printed ^[1]^[2]^[3]^[4].

To continue talking to Dosu, mention @dosu.

langgenius / dify

Knowledge base setup, mixed search requires enabling the Rerank model, but only LocalAI supports the Rerank model locally. Is there a complete example? #5865

Self Checks

Provide a description of requested docs changes