End-to-end RAG with Airbyte Cloud, Microsoft Sharepoint and Zilliz(Milvus)

aaronsteers commented 6 months ago

Summary

This goal of this task is to show uses how to use Microsoft Sharepoint as a data source for their Milvus based vector store, and to implement a short RAG demo based on the loaded data. (Note: Zilliz is a vector store built by Milvus, so you might see either name in different places)

Project Description

This issue broadly involves the following

Set up a Airbyte cloud connection using the mentioned source and destinations and load some data.
Perform RAG on data in the mentioned destination. You can choose to do RAG using OpenAI/LangChain, or any other means of your choice.

Definition of Done

Deliver documentation (Blog post, Python notebook, etc., )covering end to end steps including code snippets when applicable. When providing a python notebook, please add a "What / Why / How" blurb at the top to explain what the code is doing.

Resources to Assist

Example tutorial: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
s3 source documentation: https://docs.airbyte.com/integrations/sources/s3
Milvus documentation: https://docs.airbyte.com/integrations/destinations/milvus
LangChain Zilliz(Milvus) support: https://python.langchain.com/v0.1/docs/integrations/vectorstores/zilliz/
Sample blog post: https://airbyte.com/tutorials/chat-with-your-data-using-openai-pinecone-airbyte-and-langchain

maddy011 commented 5 months ago

Hey @aaronsteers,

Can I work on this if this isn't taken by anyone already?

Thanks!

marcosmarxm commented 5 months ago

It is yours!

marcosmarxm commented 5 months ago

@maddy011 do you need any help?

maddy011 commented 5 months ago

Yes, I was unaware that Microsoft Sharepoint requires a Microsoft 365 paid subscription account and the personal plan is quite expensive. I am figuring out if I can complete the work without having to buy the subscription. Is it possible to get a Sharepoint trial tenant id from your side?

Sorry for the delay.

marcosmarxm commented 5 months ago

@maddy011 please ping me on Slack!

marcosmarxm commented 5 months ago

@maddy011 do you need any help here?

gupta-arpan commented 4 months ago

Hi @marcosmarxm can i work on this issue too as i can see @maddy011 is inactive.

bala-ceg commented 4 months ago

I would like to take this issue, just in case if it is available

Harmaton commented 4 months ago

I would like to work on this issue if available as well.

marcosmarxm commented 4 months ago

@Harmaton all yours. @bala-ceg you have other issue assigned to you. @gupta-arpan I assigned other one to you early today.

gupta-arpan commented 4 months ago

@marcosmarxm I think you are confused with some other user as I cannot see any issue assigned to me right now.

btkcodedev commented 4 months ago

Hi, since the hackathon is reaching its end, I would like to work on this if the assigned user is busy. Cc: @marcosmarxm @aaronsteers

Harmaton commented 4 months ago

Relax my friend @btkcodedev , the task is in good hands

Harmaton commented 4 months ago

Blog Post .Please review it @aaronsteers

bleonard commented 4 months ago

Hi @Harmaton, we will take a look, thanks.

airbytehq / PyAirbyte-Hackathon