CodeForPhilly / cfp-data-pipeline

7 stars 3 forks source link

Why was the snowflake tap failing to find schemas? #32

Closed machow closed 3 years ago

machow commented 3 years ago

Leaving here for posterity. Here is the slack thread discussing issue. The snowflake tap kept erroring saying it couldn't CREATE SCHEMA, but the tap docs said it only creates schemas if they're missing.

Ended up connecting with sqlalchemy to see what schemas it thought were in the database, using code below.

from sqlalchemy import create_engine, inspect
from snowflake.sqlalchemy import URL, TIMESTAMP_NTZ
from dotenv import dotenv_values
config = dotenv_values('.env-staging')
config = {k.split('_')[-1].lower(): v for k, v in config.items() if 'SNOWFLAKE' in k}
url = URL(
                user=config["username"],
                password=config["password"],
                account=config["account"],
                database=config["database"],
                role=config["role"],
                warehouse=config["warehouse"]
)
engine = create_engine(url)
inspector = inspect(engine)

In the end, we needed to grant the privileges for database USAGE (not just schema usage).