add rerank model success for LocalAI, but no show in system

edisonzf2020 commented 6 months ago

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.8

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

add rerank model mxbai-rerank-large-v1 success.
can not display in system .

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

dosubot[bot] commented 6 months ago

To address the issue with the "mxbai-rerank-large-v1" model not showing up in the Dify system after adding it to your self-hosted LocalAI, please ensure the following steps have been correctly executed:

Model and Configuration: Confirm that the model is saved in the models directory and a correct YAML configuration file is created for it within the same directory.
Restart LocalAI: Make sure LocalAI has been restarted to load the new model configuration.
Dify Integration: In Dify, navigate to Settings > Model Providers > LocalAI and add the new model with its details to ensure it's integrated properly.

Check for these potential issues:

Model Compatibility: The model should be compatible with the ggml format and supported by Dify.
Network Configuration: Verify that the Dify container can access the LocalAI endpoint. Use the host IP address or a LAN IP address instead of localhost.

If the model still doesn't appear, review the integration process in Dify, especially the updateModelList function, to ensure the model list is refreshed correctly. Also, look out for any error messages during the process, as they can provide specific clues to the problem.

For a detailed guide on adding models to LocalAI and troubleshooting, refer to the LocalAI model configuration documentation.

If after these checks the issue persists, please share more details about the steps you've taken and any error messages encountered for further assistance.

#### Details

**Sources** - [en/tutorials/model-configuration/localai.md](https://github.com/langgenius/dify-docs/blob/main/en/tutorials/model-configuration/localai.md) - [api/core/model_runtime/model_providers/localai/localai.yaml](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/localai/localai.yaml) - [web/app/components/header/account-setting/model-provider-page/system-model-selector/index.tsx](https://github.com/langgenius/dify/blob/main/web/app/components/header/account-setting/model-provider-page/system-model-selector/index.tsx) - [api/core/model_runtime/model_providers/localai/rerank/rerank.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/localai/rerank/rerank.py) - [en/getting-started/readme/specifications-and-technical-features.md](https://github.com/langgenius/dify-docs/blob/main/en/getting-started/readme/specifications-and-technical-features.md) - [api/tests/integration_tests/model_runtime/localai/test_rerank.py](https://github.com/langgenius/dify/blob/main/api/tests/integration_tests/model_runtime/localai/test_rerank.py) - [zh_CN/learn-more/extended-reading/retrieval-augment/rerank.md](https://github.com/langgenius/dify-docs/blob/main/zh_CN/learn-more/extended-reading/retrieval-augment/rerank.md)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

gakugaku commented 6 months ago

After configuring LocalAI ReRank, running SELECT * FROM provider_models on the Dify DB returns the following result, indicating that it is configured correctly.

... | ...  | localai       | <<model_name>> | reranking       | {"server_url": "http://xxx.xxx.xxx.xxx/v1"} | t        | 2024-05-17 17:10:23 | 2024-05-23 07:26:34.66699

However, it does not show up in the model provider settings and cannot be used as a model.

It seems like the API endpoint console/api/workspaces/current/models/model-types/rerank is not functioning with the registered LocalAI Reranker.

Request: http://xxx/console/api/workspaces/current/models/model-types/rerank Response:

{data: [{provider: "nvidia", label: {zh_Hans: "API Catalog", en_US: "API Catalog"},…},…]}
0: {provider: "nvidia", label: {zh_Hans: "API Catalog", en_US: "API Catalog"},…}
1: {provider: "cohere", label: {zh_Hans: "Cohere", en_US: "Cohere"},…}
2: {provider: "jina", label: {zh_Hans: "Jina", en_US: "Jina"},…}

Environment:

Dify version: 0.6.8
LocalAI version: 2.15.0

Sarmingsteiner commented 6 months ago

@edisonzf2020 : Could you please point me out how you managed to install the reranking model mxbai-rerank-large-v1 in LocalAI?

@gakugaku : Which reranking model did you install in LocalAI and how did you manage it? Thank you in advance.

gakugaku commented 6 months ago

@Sarmingsteiner This issue has been resolved in #4652, and it should work correctly in the latest version.

Here is how I managed to install a reranking model in LocalAI. Example for Docker and Japanese Reranker model.

LocalAI Configuration

compose.yaml

services:
  local-ai:
    image: localai/localai:v2.16.0
    environment:
      MODELS_PATH: /models
    volumes:
      - ./data/LocalAI/models:/models:cached

./data/LocalAI/models/reranker.yaml You can find more details in the 📈 Reranker | LocalAI documentation.

name: japanese-reranker
backend: rerankers
parameters:
  model: hotchpotch/japanese-reranker-cross-encoder-small-v1

When you start and run this configuration, the model will be downloaded automatically.

Dify Configuration

Select the LocalAI provider and configure it as follows:

Model Type: Rerank
Model Name: In the example above, it would be japanese-reranker
Server URL: http://<your-server>/v1 (make sure to include /v1)

langgenius / dify