Dynamic Selection of Knowledge Datasets in Knowledge Retrieval Nodes

[Nazza77]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Pleas do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

While developing a local ChatBot with long-term memory capabilities, I encountered a significant limitation concerning the management of knowledge datasets. Each user must have a distinct dataset to maintain privacy and the relevance of interactions.

Currently, the selection of the dataset in the "Knowledge Retrieval" node is static and manual, which makes it impossible to automate the system for multiple users.

2. Describe the feature you'd like to see

I propose implementing a feature that allows dynamic selection of the "Knowledge Dataset" within the "Knowledge Retrieval" node.

This feature would involve using a variable, such as the knowledge dataset's ID, to automatically select the corresponding dataset. This would enable the node to load and use the correct set of knowledge specific to each user during interactions.

3. How will this feature improve your workflow or experience?

This feature would transform the capability of the ChatBot to operate coherently and personally with multiple users, ensuring that each interaction is informed by the correct context and secure personal data. Moreover, it would allow the product to scale without compromising data privacy or the relevance of the responses provided, which are essential for maintaining a high-quality user experience and compliance with privacy standards.

4. Additional context or comments

While developing a local ChatBot with long-term memory capabilities, I encountered a significant limitation concerning the management of knowledge datasets. Each user must have a distinct dataset to maintain privacy and the relevance of interactions. Currently, the selection of the dataset in the Knowledge Retrieval node is static and manual, which makes it impossible to automate the system for multiple users.

I would also like to extend my gratitude to the Dify team and the surrounding community for your hard work and dedication. Your commitment to developing such a high-quality solution is truly commendable. Keep up the great work! 😁

5. Can you help us with this feature?

• [x] I am interested in contributing to this feature.

[nathubs]

In many situation,it seems very useful to dynamic Selection of Knowledge Datasets in Knowledge Retrieval Nodes

[gnoviawan]

really need this feature! my use case is that i have a generator workflow to create an article or e-book for user, book will be have a long page, after the content generated from my workflow i will create a new dataset dedicated for the user, after that user will be able to brainstorm or chat again based on the konwledge that provided to the bot which is their e-book from the generator result, this will be on another workflow let's call it brainstorm workflow, on this workflow i need to assign the dataset based on who is the user calling the workflow , so reusable workflow API for many user with dynamic dataset using variable would be needed

[zawilliams]

Is this on the roadmap at all? This would be incredibly useful.

[weipanux]

I encounter a need for this dynamic selection/adding of knowledge datasets in Knowledge Retrieval Nodes too. My app create unique datasets after they registered. The challenge i have with Dify is that the Knowledge Retrieval Nodes right now doesn't allow me to add the newly created datasets in runtime. Currently it seems we have to manually add them in the Knowledge Retrieval Nodes, which is not scalable, not automatic.