RAG on Snowflake with Airbyte source data

[aaronsteers]

Summary

Build a RAG solution running on 100% Snowflake-managed infrastructure, using Airbyte.

Project Description

This broadly involves following steps:

• Move some data into Snowflake
• Build a RAG demo in Snowflake, using Snowflake infra and Snowflake Cortex functions. Cortex allows you to vectorize data and perform vector operations like cosine similarity search

Deliverable

• Documentation/Readme outlining end to end steps involved in building the RAG solution

Resources to Assist

• Snowflake Cortex: https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview

[Harmaton]

I would love to work on this topic, Thank you :)

[marcosmarxm]

Assigned to you!

[Harmaton]

@marcosmarxm I have completed my documentation but I am using fake for the embedding model so i cant really get responses, can you provide me with a temporary openai api key just for the purpose of getting the responses in the blog ?

[Harmaton]

Here is my documentation , waiting for the api key to finalize. Thank you!

[marcosmarxm]

Thanks @Harmaton let wait for Bindi to check the API keys.

[bindipankhudi]

@Harmaton were you able to get a key from @marcosmarxm?

[Harmaton]

No not yet @bindipankhudi . However, I have completed the blog waitimg for review. :)

[bindipankhudi]

sent you a key in DM, @Harmaton. I will look at the blog post.

[bindipankhudi]

@Harmaton - actually for this issue, you can keep using fake embedding and vectorize data using Cortex function, so basically overwrite the embedding column in the snowflake by vectorizing the document_content column using cortex'x embed function.

[bindipankhudi]

also, @Harmaton when it's complete, if you could share the blog in a google doc that would be great! :)

[Harmaton]

Okay, noted. Is it good practice to use any vector embedding function on data I passed to snowflake cortex directly (where I had to embed using fake or openai?)?

[Harmaton]

Okay, noted. Is it good practice to use vector embedding on data I passed to snowflake cortex directly (where I have to embed uaing fake or openai?)? Or should I use either?

[Harmaton]

Final Submission @bindipankhudi

[marcosmarxm]

Thanks @Harmaton I'll Bindi to take a look!

[bindipankhudi]

Thank you @Harmaton. The blog post looks good! Thank you for making the updates. Could you share a picture and a quick bio for publishing on our side? Thank you! :)

[Harmaton]

BIO + IMAGE -> HARMATON @bindipankhudi