Illustrate storing vector data into Snowflake using PyAirbyte

[bindipankhudi]

Summary

Show users how they can load Github data into snowflake using PyAirbyte, followed by prepping the data for vector search.

Description

This task involves the following

• Load issues data from Github using PyAirbyte
• Use PyAibyte's to_documents to convert each issue into documents. (Example of how to do this in resources below)
• Use Snowflake Python Connector to write Github issues into snowflake. Make sure to add new column document_content corresponding to each issue
• Once in Snowflake, use Snowflake Cortex's embed function (documented below) to create a new embedding column for document_content
• Perform QnA by doing similarity search on the data from embedding column.

Resources

• PyAirbyte Github tutorial: https://github.com/airbytehq/quickstarts/blob/main/pyairbyte_notebooks/PyAirbyte_Shopify_Demo.ipynb
• PyAirbyte to_documents tutorial: https://github.com/airbytehq/quickstarts/blob/main/pyairbyte_notebooks/PyAirbyte_Document_Creation_RAG_with_Langchain_Demo.ipynb
• Snowflake Cortex embed function: https://docs.snowflake.com/en/sql-reference/functions/embed_text-snowflake-cortex
• Cortex similarity search example: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
• PyAirbyte Documentaton: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started
• PyAirbyte API Reference: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started#api-reference

[fvgm-spec]

Hi @bindipankhudi could you please assign this one to me?

[marcosmarxm]

It is yours @fvgm-spec

[fvgm-spec]

Hi @bindipankhudi I am trying to connect a trial snowflake account using the Python snowflake.connector, I am following your sample tutorial and setting the connecting with the below configuration getting an error connection, I suspect that is related to the SNOWFLAKE_HOST

Could you provide a guideline on connecting Snowflake using the Python connector?

[bindipankhudi]

Hi Felix, try using the account name as the host. For example, instead of xyz.snowflakecomputing.com, just use xyz. The name SNOWFLAKE_HOST is a bit confusing. The actually account name should be passed here.

If ur having trouble locating the account name, look at the browser URL when you log in to snowflake.

Bindi Pankhudi

Engineering Lead, AI/LLM

GitHub https://github.com/bindipankhudi | LinkedIn https://www.linkedin.com/in/pankhudisinha/

We're hiring, come work with me! https://airbyte.io/careers [image: 🚀]

On Sat, Jun 8, 2024 at 8:08 AM Felix Gutierrez @.***> wrote:

Hi @bindipankhudi https://github.com/bindipankhudi I am trying to connect a trial snowflake account using the Python snowflake.connector, I am following your sample tutorial https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb and setting the connecting with the below configuration getting an error connection, I suspect that is related to the SNOWFLAKE_HOST

image.png (view on web) https://github.com/airbytehq/PyAirbyte-Hackathon/assets/60470663/53d9947c-bc32-4926-ba73-6f9ac67482bc

Could you provide a guideline on connecting Snowflake using the Python connector?

— Reply to this email directly, view it on GitHub https://github.com/airbytehq/PyAirbyte-Hackathon/issues/28#issuecomment-2156070191, or unsubscribe https://github.com/notifications/unsubscribe-auth/BF2EJSQYADW54BBQS55FTK3ZGMNALAVCNFSM6AAAAABIZWRXGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWGA3TAMJZGE . You are receiving this because you were mentioned.Message ID: @.***>

[fvgm-spec]

Hi @bindipankhudi thanks so much for your quick response the URL of my snowflake instance is a trial one, hence it does not have the snowflakecomputing.com part, mine is https://app.snowflake.com/ibqtudo/nl03202 so I have tried to set separately ibqtudo and nl03202 as the HOST but none of them works. Maybe it could be because it is a trial account?

[bindipankhudi]

Maybe try this to get the account name:

https://docs.snowflake.com/en/sql-reference/functions/current_account_name

Bindi Pankhudi

Engineering Lead, AI/LLM

GitHub https://github.com/bindipankhudi | LinkedIn https://www.linkedin.com/in/pankhudisinha/

We're hiring, come work with me! https://airbyte.io/careers [image: 🚀]

On Sat, Jun 8, 2024 at 8:58 AM Felix Gutierrez @.***> wrote:

Hi @bindipankhudi https://github.com/bindipankhudi thanks so much for your quick response the URL of my snowflake instance is a trial one, hence it does not have the snowflakecomputing.com part, mine is https://app.snowflake.com/ibqtudo/nl03202 so I have tried to set separately ibqtudo and nl03202 as the HOST but none of them works. Maybe it could be because it is a trial account?

image.png (view on web) https://github.com/airbytehq/PyAirbyte-Hackathon/assets/60470663/47d7e1e8-894e-41e0-8314-3d580ba4db50

— Reply to this email directly, view it on GitHub https://github.com/airbytehq/PyAirbyte-Hackathon/issues/28#issuecomment-2156085712, or unsubscribe https://github.com/notifications/unsubscribe-auth/BF2EJSVM5AR66KMXZA4OT3LZGMS2RAVCNFSM6AAAAABIZWRXGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWGA4DKNZRGI . You are receiving this because you were mentioned.Message ID: @.***>

[marcosmarxm]

@fvgm-spec let me know if the troubleshooting session was helpful and got you unblocked.

[fvgm-spec]

Hey @marcosmarxm sure it was beneficial in handling the output from the to_documents method, so I think I will make it with a little more work. I will let you know!

[fvgm-spec]

Hey @marcosmarxm once I have spent long hours troubleshooting this, I am quitting it :(, The last thing I did was to store the issues.content into a list so I could load it into the snowflake table:

Then loading the content strings stored in the list, no success:

It seems to be related to special characters in the strings that contain the content:

ProgrammingError: 001003 (42000): SQL compilation error:
parse error line 8 at position 5 near '<EOF>'.
syntax error line 1 at position 64 unexpected '# '.
parse error line 3 at position 7 near '10'.

This is the snowflake table, and I am leaving available the notebook that contains all the code I was working on

[fvgm-spec]

Hi @marcosmarxm could you please unassign me this one, as I am not able to continue working on it? Thanks. Could I be assigned to this one? https://github.com/orgs/airbytehq/projects/75/views/4?pane=issue&itemId=63202677

[gupta-arpan]

Hi @marcosmarxm @bindipankhudi can you assign me this issue i would like to work on this, Thanks.

[marcosmarxm]

@gupta-arpan all yours!

[gupta-arpan]

@bindipankhudi I have made a PR please review it.

[bindipankhudi]

Thank you @gupta-arpan! I will review this.

[bindipankhudi]

Reviewed and merged! Thank you @gupta-arpan for the good work! :)