Empty retriever_resources Node in API Response

[yjc980121]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.11.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Endpoint: /v1/messages

Description: When making a GET request to the /v1/messages endpoint with the following parameters:

user=abc-123

conversation_id=

and including the Authorization: Bearer {api_key} header, the response contains an empty retriever_resources node. This behavior is inconsistent with the documentation, which provides an example response that includes a non-empty retriever_resources node.

Expected Behavior: The retriever_resources node in the response should contain the expected data, allowing the caller to display the source of the referenced information when viewing the conversation history.

Actual Behavior: The retriever_resources node in the response is empty, which prevents the caller from displaying the source of the referenced information in the conversation history.

Steps to Reproduce:

Make a GET request to /v1/messages with the query parameters user=abc-123 and conversation_id=.

Include the Authorization: Bearer {api_key} header.

Observe the response, specifically the retriever_resources node.

Impact: This issue prevents the caller from displaying the source of the referenced information in the conversation history, which is a critical feature for the application.

Request: Please investigate why the retriever_resources node is empty and update the API to return the expected data as described in the documentation.

✔️ Expected Behavior

{
  "limit": 20,
  "has_more": false,
  "data": [
    {
      "id": "a076a87f-31e5-48dc-b452-0061adbbc922",
      "conversation_id": "cd78daf6-f9e4-4463-9ff2-54257230a0ce",
      "inputs": {
        "name": "dify"
      },
      "query": "iphone 13 pro",
      "answer": "The iPhone 13 Pro, released on September 24, 2021, features a 6.1-inch display with a resolution of 1170 x 2532. It is equipped with a Hexa-core (2x3.23 GHz Avalanche + 4x1.82 GHz Blizzard) processor, 6 GB of RAM, and offers storage options of 128 GB, 256 GB, 512 GB, and 1 TB. The camera is 12 MP, the battery capacity is 3095 mAh, and it runs on iOS 15.",
      "message_files": [],
      "feedback": null,
      "retriever_resources": [
        {
          "position": 1,
          "dataset_id": "101b4c97-fc2e-463c-90b1-5261a4cdcafb",
          "dataset_name": "iPhone",
          "document_id": "8dd1ad74-0b5f-4175-b735-7d98bbbb4e00",
          "document_name": "iPhone List",
          "segment_id": "ed599c7f-2766-4294-9d1d-e5235a61270a",
          "score": 0.98457545,
          "content": "\"Model\",\"Release Date\",\"Display Size\",\"Resolution\",\"Processor\",\"RAM\",\"Storage\",\"Camera\",\"Battery\",\"Operating System\"\n\"iPhone 13 Pro Max\",\"September 24, 2021\",\"6.7 inch\",\"1284 x 2778\",\"Hexa-core (2x3.23 GHz Avalanche + 4x1.82 GHz Blizzard)\",\"6 GB\",\"128, 256, 512 GB, 1TB\",\"12 MP\",\"4352 mAh\",\"iOS 15\""
        }
      ],
      "created_at": 1705569239
    }
  ]
}

❌ Actual Behavior

{
  "limit": 20,
  "has_more": false,
  "data": [
    {
      "id": "a076a87f-31e5-48dc-b452-0061adbbc922",
      "conversation_id": "cd78daf6-f9e4-4463-9ff2-54257230a0ce",
      "inputs": {
        "name": "dify"
      },
      "query": "iphone 13 pro",
      "answer": "The iPhone 13 Pro, released on September 24, 2021, features a 6.1-inch display with a resolution of 1170 x 2532. It is equipped with a Hexa-core (2x3.23 GHz Avalanche + 4x1.82 GHz Blizzard) processor, 6 GB of RAM, and offers storage options of 128 GB, 256 GB, 512 GB, and 1 TB. The camera is 12 MP, the battery capacity is 3095 mAh, and it runs on iOS 15.",
      "message_files": [],
      "feedback": null,
      "retriever_resources": [],
      "created_at": 1705569239
    }
  ]
}

[dosubot[bot]]

@yjc980121 Hello there! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's work together to resolve this issue.

To address the issue of the empty retriever_resources node in the API response, you should verify the following conditions and configurations:

1. Configuration Check: Ensure that the retriever_resource configuration is enabled. The configuration should include "retriever_resource": {"enabled": True} to ensure that retriever resources are included in the response ^[1].

2. Context Fetching Conditions: The retriever_resources node is populated when the _fetch_context method yields a RunRetrieverResourceEvent with non-empty retriever_resources. This requires:

   • node_data.context.enabled to be True.
   • node_data.context.variable_selector to be set.
   • The context_value_variable retrieved from the variable_pool to be an ArraySegment.
   • Each item in the context_value_variable.value should contain a "content" key, and the _convert_to_original_retriever_resource method should return a non-None value for each item ^[2].

3. Error Handling: Check for any errors during the retrieval process. The code includes error handling that logs warnings and returns a failed status if a KnowledgeRetrievalNodeError occurs ^[3].

4. Filtering Logic: Ensure that the filtering logic does not exclude relevant retriever resources. This includes checking dataset availability, score thresholds, and segment filtering ^[4].

By ensuring these conditions and configurations are correctly set, the retriever_resources node should be populated as expected in the API response.

To continue talking to Dosu, mention @dosu.

[crazywoola]

Please check if this retriever_resource switch is enabled.