the auto-created index on Milvus is not working

bowenliang123 commented 8 months ago

Self Checks

[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] Pleas do not modify this template :) and fill in all the required fields.

Dify version

main/0.5.6

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

use milvus 2.3.9 as vector backend
upload document to knowledge library

✔️ Expected Behavior

No response

❌ Actual Behavior

it's not able to be queried on or used

it's not working when deleting docs from index , when disabling docs. logs from the worker.

[2024-02-27 02:49:45,540: ERROR/MainProcess] Cleaned document when document deleted failed
Traceback (most recent call last):
File "/Users/bw/dev/dify/api/tasks/clean_document_task.py", line 36, in clean_document_task
index_processor.clean(dataset, index_node_ids)
File "/Users/bw/dev/dify/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 68, in clean
vector.delete_by_ids(node_ids)
File "/Users/bw/dev/dify/api/core/rag/datasource/vdb/vector_factory.py", line 141, in delete_by_ids
self._vector_processor.delete_by_ids(ids)
File "/Users/bw/dev/dify/api/core/rag/datasource/vdb/milvus/milvus_vector.py", line 127, in delete_by_ids
self._client.delete(collection_name=self._collection_name, pks=doc_ids)
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 413, in delete
raise ex from ex
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 409, in delete
res = conn.delete(collection_name, expr, timeout=timeout, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/decorators.py", line 127, in handler
raise e from e
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/decorators.py", line 123, in handler
return func(*args, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/decorators.py", line 162, in handler
return func(self, *args, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/decorators.py", line 102, in handler
raise e from e
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/decorators.py", line 68, in handler
return func(*args, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 604, in delete
raise err from err
File "/opt/homebrew/Caskroom/miniforge/base/envs/dify/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 600, in delete
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create expr plan, expr =  id in [47102f94-24ed-40bb-a443-4501c256007f,42cec4f7-431f-4653-8bc9-f575219fa1c1,46d7afe5-1590-45df-9460-376a1f454e32])>

dosubot[bot] commented 8 months ago

🤖

From the error message you provided, it seems like the issue is related to the delete_by_ids method in the milvus_vector.py file. The error message indicates that the Milvus client failed to create an expression plan for the deletion operation. This could be due to a variety of reasons, such as an issue with the IDs you're trying to delete, a problem with the Milvus client, or a bug in the Dify code.

Here are a few steps you can take to troubleshoot this issue:

Check the IDs: Make sure the IDs you're trying to delete actually exist in the Milvus collection. You can do this by querying the collection with the IDs before trying to delete them. If the IDs don't exist, that could be the cause of the error.
Update the Milvus client: There might be a bug in the version of the Milvus client you're using that's causing this issue. Try updating to the latest version of the Milvus client and see if that resolves the issue.
Debug the delete_by_ids method: If the above steps don't resolve the issue, you might need to debug the delete_by_ids method in the milvus_vector.py file. You can do this by adding print statements in the method to print out the values of the variables and the execution flow. This might help you identify where exactly the issue is occurring.

If none of these steps resolve the issue, it would be helpful if you could provide more information about your setup and the steps you're taking that lead to this error. Specifically, it would be helpful to know:

The exact steps you're taking that lead to this error.
The full stack trace of the error.
The version of the Milvus client you're using.
Any other relevant information about your setup.

This information will help us better understand the issue and provide a more accurate solution.

For more information about the delete_by_ids method and how it interacts with the Milvus client, you can refer to the Milvus Python SDK documentation.

I hope this helps! Let me know if you have any other questions or if there's anything else I can do to assist you.

Sources

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

crazywoola commented 8 months ago

Fixed in https://github.com/langgenius/dify/pull/2573

langgenius / dify