fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
853 stars 72 forks source link

Unable to override the database when using the dbt-trino connector #775

Open radium226 opened 1 year ago

radium226 commented 1 year ago

Describe the bug I'm not able to override the database when using fal-dbt with the dbt-trino connector.

Your environment

How to reproduce In my dbt_project.yml, I have this in models.metabase_reports (note that the default database is not lakehouse but metabase):

experimental:
        +database: lakehouse
        +schema: lakehouse_gold

In the models/experimental/experimental__experimental.py file, I have:

def model(dbt, session):
    users_df = dbt.ref("all_users")

    first_name_df = users_df["first_name"].drop_duplicates()

    return first_name_df

When I run dbt run --select=experimental__expertimental.

Expected behavior A table should be created in the lakehouse.lakehouse_gold schema within Trino.

Actual behavior

14:38 $ ./run-locally.sh -p metabase dbt run --select=experimental__experimental
13:38:24  Running with dbt=1.4.0
13:38:24  Unable to do partial parsing because of a version mismatch
13:38:25  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.elementary
13:38:25  Found 53 models, 30 tests, 0 snapshots, 0 analyses, 661 macros, 0 operations, 0 seed files, 26 sources, 0 exposures, 0 metrics
13:38:25  
13:38:28  Concurrency: 1 threads (target='metabase_with_fal')
13:38:28  
13:38:28  1 of 1 START python table model lakehouse.lakehouse_gold.experimental__experimental  [RUN]
13:38:38  Unhandled error while executing target/run/metabase_reports/models/experimental/experimental__experimental.py
(trino.exceptions.TrinoUserError) TrinoUserError(type=USER_ERROR, name=NOT_FOUND, message="Schema lakehouse_gold not found", query_id=20230306_133838_00374_eem4m)
[SQL: 
CREATE TABLE lakehouse_gold.__dbt_fal_temp_experimental__experimental (
    first_name VARCHAR
)

]
(Background on this error at: https://sqlalche.me/e/14/f405)
13:38:38  1 of 1 ERROR creating python table model lakehouse.lakehouse_gold.experimental__experimental  [ERROR in 9.78s]
13:38:38  
13:38:38  Finished running 1 table model in 0 hours 0 minutes and 12.84 seconds (12.84s).
13:38:38  
13:38:38  Completed with 1 error and 0 warnings:
13:38:38  
13:38:38  (trino.exceptions.TrinoUserError) TrinoUserError(type=USER_ERROR, name=NOT_FOUND, message="Schema lakehouse_gold not found", query_id=20230306_133838_00374_eem4m)
13:38:38  [SQL: 
13:38:38  CREATE TABLE lakehouse_gold.__dbt_fal_temp_experimental__experimental (
13:38:38    first_name VARCHAR
13:38:38  )
13:38:38  
13:38:38  ]
13:38:38  (Background on this error at: https://sqlalche.me/e/14/f405)
13:38:38  
13:38:38  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

The issue it that the database is not prepended to the table FQN (we only have the schema).