RAG on Snowflake with Airbyte source data

airbytehq / PyAirbyte-Hackathon

Tasks for PyAirbyte Hackathon June 2024

0 stars 2 forks source link

RAG on Snowflake with Airbyte source data #2

Closed aaronsteers closed 2 weeks ago

aaronsteers commented 1 month ago

Summary

Build a RAG solution running on 100% Snowflake-managed infrastructure, using Airbyte.

Project Description

This broadly involves following steps:

Move some data into Snowflake
Build a RAG demo in Snowflake, using Snowflake infra and Snowflake Cortex functions. Cortex allows you to vectorize data and perform vector operations like cosine similarity search

Deliverable

Documentation/Readme outlining end to end steps involved in building the RAG solution

Resources to Assist

Snowflake Cortex: https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview

Harmaton commented 4 weeks ago

I would love to work on this topic, Thank you :)

marcosmarxm commented 4 weeks ago

Assigned to you!

Harmaton commented 3 weeks ago

@marcosmarxm I have completed my documentation but I am using fake for the embedding model so i cant really get responses, can you provide me with a temporary openai api key just for the purpose of getting the responses in the blog ?

Harmaton commented 3 weeks ago

Here is my documentation , waiting for the api key to finalize. Thank you!

marcosmarxm commented 3 weeks ago

Thanks @Harmaton let wait for Bindi to check the API keys.

bindipankhudi commented 2 weeks ago

@Harmaton were you able to get a key from @marcosmarxm?

Harmaton commented 2 weeks ago

No not yet @bindipankhudi . However, I have completed the blog waitimg for review. :)

bindipankhudi commented 2 weeks ago

sent you a key in DM, @Harmaton. I will look at the blog post.

bindipankhudi commented 2 weeks ago

@Harmaton - actually for this issue, you can keep using fake embedding and vectorize data using Cortex function, so basically overwrite the embedding column in the snowflake by vectorizing the document_content column using cortex'x embed function.

bindipankhudi commented 2 weeks ago

also, @Harmaton when it's complete, if you could share the blog in a google doc that would be great! :)

Harmaton commented 2 weeks ago

Okay, noted. Is it good practice to use any vector embedding function on data I passed to snowflake cortex directly (where I had to embed using fake or openai?)?

Harmaton commented 2 weeks ago

Okay, noted. Is it good practice to use vector embedding on data I passed to snowflake cortex directly (where I have to embed uaing fake or openai?)? Or should I use either?

Harmaton commented 2 weeks ago

Final Submission @bindipankhudi

marcosmarxm commented 2 weeks ago

Thanks @Harmaton I'll Bindi to take a look!

bindipankhudi commented 2 weeks ago

Thank you @Harmaton. The blog post looks good! Thank you for making the updates. Could you share a picture and a quick bio for publishing on our side? Thank you! :)

Harmaton commented 2 weeks ago

BIO + IMAGE -> HARMATON @bindipankhudi