❄️ Snowflake connector

[mattigrthr]

User story

As a user, I quickly want to connect my Snowflake data warehouse with Kuwala to start applying transformations. I only want to put in my credentials and establish the connection. Once connected, I want to see the database schema to see all available tables. For every existing table, I want to see a preview of the data and the column types.

Acceptance criteria

• [x] The user can establish a connection to their Snowflake data warehouse by providing the necessary credentials.
• [x] The user can retrieve the database schema and explore all available tables.
• [x] The user can preview the data in the table and its data types.
• [x] The dbt project in the backend is automatically set up

Non-functional requirements

• [x] Connecting to the Snowflake takes less than 5 seconds.
• [x] Loading the list of all available tables takes less than 5 seconds.
• [x] Loading the preview of a table takes less than 5 seconds.
• [x] In the preview of a table, up to 200 columns are displayed instantly.
• [x] In the preview of a table, up to 300 rows are displayed instantly.

Resources and technical details

To establish the connection and interaction with Snowflake, we can use snowflake-connector-python.

The automatic dbt project setup should be integrated along the lines of #83 after it is merged. Dbt has an adapter for Snowflake as well.

API routes

To be able to select Snowflake, you need to add it to the data catalog.

• [x] Test connection to data source
• [x] Save credentials for data source
• [x] Get hierarchy of database (schema, category, tables)
• [x] Get preview of data in table

Designs

The designs are in the same manner as for #80.

[EteimZ]

Hi @mattigrthr I will love to work on this issue.

[mattigrthr]

Awesome! 🙌🏽 Please create a fork and open the PR from there to the main repo (see our contributing guidelines). You can also create the PR as a draft early on before it's ready for review. Then it's easier to follow along in between. 🤓

[mattigrthr]

Closed with #116

[mattigrthr]

Only merged into dev atm. Reopening and closing when in master.