Sentiment analysis using Airbyte Cloud, Google sheets, and Snowflake Cortex

bindipankhudi commented 5 months ago

Summary

This goal of this task is to show users how they load some dummy user review data from google sheets intoSnowflake Cortex based vector store, and perform sentiment analysis using Snowflake Cortex's sentiment function.

Project Description

This issue broadly involves the following

Set up a Airbyte cloud connection using the mentioned source and destinations. (Feel free to use dummy reviews data like restaurant, or product reviews)
Do sentiment analysis on data in Snowflake.

Definition of Done

Deliver documentation (Blog post, Python notebook, etc., )covering end to end steps including code snippets when applicable. When providing a python notebook, please add a "What / Why / How" blurb at the top to explain what the code is doing.

Resources to Assist

Example tutorial: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
Google sheets documentation: https://docs.airbyte.com/integrations/sources/google-sheets
Snowflake cortex documentation: https://docs.airbyte.com/integrations/destinations/snowflake-cortex
Snowflake Cortex functions: https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview
Sample blog post: https://airbyte.com/blog/build-end-to-end-rag-applications-using-airbyte-and-snowflake-cortex

bala-ceg commented 5 months ago

Hi @aaronsteers, Can i work on this issue, while the issue #13 is sorted out?

marcosmarxm commented 5 months ago

It is yours @bala-ceg

bala-ceg commented 5 months ago

Hi @bindipankhudi, Can you please confirm whether the Jupyter notebook tutorial like this is sufficient for this issue

bindipankhudi commented 5 months ago

Hi @bala-ceg, yep, what you shares is sufficient. Please make sure to add enough description in the notebook, so it's self-sufficient and easy to follow.

bala-ceg commented 5 months ago

@bindipankhudi fyi, requested credentials in dm.

bala-ceg commented 5 months ago

Hi @bindipankhudi, Does this issue requires use of snowflake (not snowflake cortex) to run batch predictions? Please clarify

bala-ceg commented 5 months ago

This is initial google colab notebook draft for sentiment analysis using snowflake https://colab.research.google.com/drive/1fwCL2dvCGAsO9SZQ_wWEEVVDjzmGfc0u?usp=sharing

bindipankhudi commented 5 months ago

Thank you @bala-ceg! we will review it soon.

bala-ceg commented 5 months ago

Thanks @bindipankhudi , looking forward to hear from you.

bindipankhudi commented 5 months ago

Hi @bala-ceg, thank you for your patience. few comments:

Before you create a PR, please add some comments on what is being done in the notebook.
Please use Cortex's sentiment analysis function to do the analysis: https://docs.snowflake.com/en/sql-reference/functions/sentiment-snowflake-cortex. You are using openAI here.
Per issue, you should use Airbyte cloud to move data into Snowflake Cortex destination. This is not clear from the colab.

bala-ceg commented 4 months ago

Hi @bindipankhudi, I am getting the below error, is there anything do i need to configure from snowflake?

ProgrammingError: 002141 (42601): SQL compilation error:
Unknown user-defined function SNOWFLAKE.CORTEX.COMPLETE

https://colab.research.google.com/drive/1fwCL2dvCGAsO9SZQ_wWEEVVDjzmGfc0u?usp=sharing

bindipankhudi commented 4 months ago

@bala-ceg you can check in snowflake to see if you have access to the complete function. Here is a screenshot from one of our snowflake accounts.

bala-ceg commented 4 months ago

Hi @bindipankhudi, I am not able to enable cortex in my snowflake account. is it possible to provide me demo/sample credentials that airbyte team has?

bindipankhudi commented 4 months ago

HI @bala-ceg, let me check with @marcosmarxm if this is something we can share.

bala-ceg commented 4 months ago

Thanks @bindipankhudi, I am waiting for this. I will close this by today or tomorrow

bindipankhudi commented 4 months ago

Hi @bala-ceg, I checked with Marcos and we decided that it's not ideal for us to share our database credentials, so seems like the most plausible option for you might be to drop this ticket and pick an new one.

ombhardwajj commented 4 months ago

Just incase the issue I suggested is not being added, and if @bala-ceg is dropping off ,I would like to give this a try so please assign me! @marcosmarxm @bindipankhudi

ombhardwajj commented 4 months ago

Hey @bala-ceg Please confirm if you've dropped off!

bala-ceg commented 4 months ago

Hi @ombhardwajj, @marcosmarxm & @bindipankhudi , let me take another run at getting cortex support to my account. If not i will unassign muself

bala-ceg commented 4 months ago

I am able to get cortex support, will create a pr within next 24 hrs

aaronsteers commented 4 months ago

Thanks, @bala-ceg. I'll check in tomorrow (Saturday) in case there's any assistance I can lend or questions I can answer.

avirajsingh7 commented 4 months ago

@bala-ceg If you are still stuck with this issue, I would like to work on this. (Already Raised Pr for all other issues assigned to me) cc: @aaronsteers, @marcosmarxm

bala-ceg commented 4 months ago

Thanks for chiming in on this issue, but i got it for now

bala-ceg commented 4 months ago

this is the draft collab notebook - https://colab.research.google.com/drive/1fwCL2dvCGAsO9SZQ_wWEEVVDjzmGfc0u?usp=sharing

bala-ceg commented 4 months ago

@bindipankhudi can you please review the PR?

bleonard commented 4 months ago

Merged https://github.com/airbytehq/quickstarts/pull/129

airbytehq / PyAirbyte-Hackathon