End-to-end RAG with Airbyte Cloud, Microsoft Sharepoint and Zilliz(Milvus)

aaronsteers commented 1 month ago

Summary

This goal of this task is to show uses how to use Microsoft Sharepoint as a data source for their Milvus based vector store, and to implement a short RAG demo based on the loaded data. (Note: Zilliz is a vector store built by Milvus, so you might see either name in different places)

Project Description

This issue broadly involves the following

Set up a Airbyte cloud connection using the mentioned source and destinations and load some data.
Perform RAG on data in the mentioned destination. You can choose to do RAG using OpenAI/LangChain, or any other means of your choice.

Definition of Done

Deliver documentation (Blog post, Python notebook, etc., )covering end to end steps including code snippets when applicable. When providing a python notebook, please add a "What / Why / How" blurb at the top to explain what the code is doing.

Resources to Assist

Example tutorial: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
s3 source documentation: https://docs.airbyte.com/integrations/sources/s3
Milvus documentation: https://docs.airbyte.com/integrations/destinations/milvus
LangChain Zilliz(Milvus) support: https://python.langchain.com/v0.1/docs/integrations/vectorstores/zilliz/
Sample blog post: https://airbyte.com/tutorials/chat-with-your-data-using-openai-pinecone-airbyte-and-langchain

maddy011 commented 3 weeks ago

Hey @aaronsteers,

Can I work on this if this isn't taken by anyone already?

Thanks!

marcosmarxm commented 3 weeks ago

It is yours!

marcosmarxm commented 2 weeks ago

@maddy011 do you need any help?

maddy011 commented 2 weeks ago

Yes, I was unaware that Microsoft Sharepoint requires a Microsoft 365 paid subscription account and the personal plan is quite expensive. I am figuring out if I can complete the work without having to buy the subscription. Is it possible to get a Sharepoint trial tenant id from your side?

Sorry for the delay.

marcosmarxm commented 2 weeks ago

@maddy011 please ping me on Slack!

marcosmarxm commented 1 week ago

@maddy011 do you need any help here?

gupta-arpan commented 1 week ago

Hi @marcosmarxm can i work on this issue too as i can see @maddy011 is inactive.

bala-ceg commented 5 days ago

I would like to take this issue, just in case if it is available

Harmaton commented 5 days ago

I would like to work on this issue if available as well.

marcosmarxm commented 4 days ago

@Harmaton all yours. @bala-ceg you have other issue assigned to you. @gupta-arpan I assigned other one to you early today.

gupta-arpan commented 4 days ago

@marcosmarxm I think you are confused with some other user as I cannot see any issue assigned to me right now.

airbytehq / PyAirbyte-Hackathon