When using multi-channel recall without configuring the rerank model and only using the weight mode, the order of the recalled segments cannot be fixed and there is a problem with the number of recalled segments

[huihuiisbest]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.16

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

The response effect is unstable (sometimes good and sometimes bad), and analysis shows that there are problems with the rearrangement after multiple recalls.

1. The recall order of multiple knowledge bases is not fixed. Sometimes the relevant fragments of knowledge base A come first, and sometimes the relevant fragments of knowledge base B come first
2. There is a problem with reordering related fragments from multiple knowledge bases: it can be observed that the sorting is often done as a whole, rather than by fragments. Expected to see multiple fragments from different knowledge bases arranged in a staggered manner.
3. The issue of the number of recalled segments. After setting the Top_K number of multiple recall strategies, theoretically the total number of fragments recalled in multiple recalls should not exceed Top_K. However, in practice, it was found during use that the number of fragments recalled in each knowledge base should not exceed Top_K.

✔️ Expected Behavior

For the same problem, the recall order of multiple knowledge bases is fixed. And different knowledge base fragments are arranged in a cross pattern based on similarity.

❌ Actual Behavior

There are issues with multi-channel recall: the same problem, the order of recall segments is not fixed, and there is no staggered sorting。

[dosubot[bot]]

Hello @huihuiisbest! I'm here to help you with bug fixes, answer your questions, and guide you on how to contribute.

I found a similar issue that was previously reported and resolved. The issue was related to the rerank_mode configuration not being respected in the app. It was closed by a pull request that fixed the problem ^[1].

To address the unstable response effect and inconsistent recall order of segments when using multi-channel recall without configuring the rerank model and only using the weight mode in Dify version 0.6.16, you can adjust the weight settings in the hybrid search mode.

Here is how you can configure the weight settings:

1. Navigate to Hybrid Search Settings: In the retrieval settings, select the hybrid search mode.
2. Adjust Weight Settings: Set custom weights for semantic priority and keyword priority. This allows you to balance between semantic search and keyword search to achieve more stable and consistent recall results.

Example configuration:

weight_settings:
  semantic_value: 0.7
  keyword_value: 0.3

This configuration gives more weight to semantic search, which can help in stabilizing the response effect and making the recall order of segments more consistent ^[2][3].

Additionally, ensure that the embedding models used in the multiple referenced knowledge bases are consistent. Inconsistent embedding models can lead to conflicts in retrieval content. Also, check if your knowledge base is using the “Economical” index mode. If so, switch it to the “High Quality” index mode to access the "Weight Score" option ^[3][2].

The default setting for the Top_K parameter in multi-channel recall strategies is 3. This parameter filters the text chunks that are most similar to the user's question. Adjusting this parameter can help manage the number of recalled segments ^[2][4][5].

To continue talking to Dosu, mention @dosu.