mindsdb / mindsdb

The platform for building AI from enterprise data
https://mindsdb.com
Other
26.72k stars 4.87k forks source link

[Manual Test] Create and sync a knowledge base from Apache Druid #10042

Closed martyna-mindsdb closed 2 days ago

martyna-mindsdb commented 6 days ago

MindsDB offers the ability to create a Knowledge Base (KB) and synchronize data from any supported data source into it. This issue provides an example of creating a Knowledge Base and automatically syncing content from Apache Druid.

The Knowledge Base will use OpenAI’s embedding model and ChromaDB as the default vector store. The end goal is to showcase an automated process where new data from Apache Druid is continuously added to the Knowledge Base using MindsDB's JOBS feature.

Steps for Contributors :checkered_flag: :shipit:

  1. Create a New Knowledge Base :new: :

    Use the CREATE KNOWLEDGE_BASE statement to establish a new KB. See examples of how to create and use a knowledge base.

  2. Integrate Apache Druid :open_file_folder: :

    Set up Apache Druid integration to allow data transfer from Apache Druid into the MindsDB environment.

  3. Automate Data Sync using JOBS :timer_clock: :

    Create a JOB that automates the process of adding new data from the Apache Druid source to the Knowledge Base.

  4. Testing and Verification :ballot_box_with_check: :

    Ensure that the data is correctly synced into the Knowledge Base. If you encounter issues, open a new issue or submit a PR to resolve the problem.

Additional Resources :books:

Deliverables :heavy_check_mark:

:loudspeaker: Additional rewards :trophy:

This issue brings :three: :zero: points for SWAG and entry into the draw for a :point_down: :


NOTE: Multiple contributors can work on this issue simultaneously without being assigned. Upon completion, share the screenshots below (if tests succeeded) or create a bug issue (if tests failed).

quest-bot[bot] commented 3 days ago

New Quest! image New Quest!

A new Quest has been launched in @mindsdb’s repo. Merge a PR that solves this issue to loot the Quest and earn your reward.


A badge has been stashed in this issue to reward the solver!

🗡 Comment @quest-bot embark to check-in for this Quest and start solving the issue. Other solvers will be notified!

⚔️ When you submit a PR, comment @quest-bot loot #10042 to link your PR to this Quest.

Questions? Check out the docs.

chandrevdw31 commented 3 days ago

Please test with existing Apache Druid credentials

Better-Boy commented 3 days ago

Raised bug - https://github.com/mindsdb/mindsdb/issues/10069

Better-Boy commented 3 days ago

Excluding the bug, the knowledge base is working fine without the last keyword.

https://github.com/user-attachments/assets/845803c2-bb37-4507-a8c9-cb5113a0910e