End-to-end RAG using Airbyte Cloud, S3 and Snowflake Cortex - Githubissues

airbytehq / PyAirbyte-Hackathon

Tasks for PyAirbyte Hackathon June 2024

0 stars 2 forks source link

End-to-end RAG using Airbyte Cloud, S3 and Snowflake Cortex #16

Closed aaronsteers closed 4 months ago

aaronsteers commented 6 months ago

Summary

This goal of this task is to show uses how to use s3 as a data source for their Snowflake Cortex based vector store, and to implement a short RAG demo based on the loaded data.

Project Description

This issue broadly involves the following

Set up a Airbyte cloud connection using the mentioned source and destinations.
Perform RAG on data in Snowflake. You can choose to do RAG using OpenAI/LangChain, or Snowflake's Cortex functions.

Definition of Done

Deliver documentation (Blog post, Python notebook, etc., )covering end to end steps including code snippets when applicable. When providing a python notebook, please add a "What / Why / How" blurb at the top to explain what the code is doing.

Resources to Assist

Example tutorial: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
s3 source documentation: https://docs.airbyte.com/integrations/sources/s3
Snowflake cortex documentation: https://docs.airbyte.com/integrations/destinations/snowflake-cortex
Snowflake Cortex functions: https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview
Sample blog post: https://airbyte.com/blog/build-end-to-end-rag-applications-using-airbyte-and-snowflake-cortex

ombhardwajj commented 5 months ago

Please assign it to me, I would like to work upon it!

marcosmarxm commented 5 months ago

It is yours!

avirajsingh7 commented 5 months ago

@ombhardwajj if you have not started working on this issue, Can I take this one?, I have already worked on almost similar issue

ombhardwajj commented 5 months ago

@avirajsingh7 Alright! You can take this!!

avirajsingh7 commented 5 months ago

@ombhardwajj thanks, @bindipankhudi assign me this one, already raised pr for other issues

Can you please review the blog Preview?

bindipankhudi commented 5 months ago

Sounds good! assigned to you @avirajsingh7. Once PR is ready, please add the link for PR here as well. That will be super helpful.

avirajsingh7 commented 5 months ago

@bindipankhudi here's blog in docx format

bindipankhudi commented 5 months ago

@avirajsingh7 blog looks great! We will publish it on Wednesday (PST). Will update here.

avirajsingh7 commented 4 months ago

Hey @bindipankhudi , any updates on this?

bindipankhudi commented 4 months ago

Hi @avirajsingh7 , this has been published! : https://airbyte.com/blog/end-to-end-rag-using-airbyte-cloud-s3-and-snowflake-cortex cc @marcosmarxm