Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

Failed on the second run #78

Closed Gatsby-Lee closed 2 years ago

Gatsby-Lee commented 2 years ago

Hi,

This is my model sql.

{{
    config(
        materialized="view",
    )
}}

SELECT org_feature_name
FROM test.org_feature_v1
GROUP BY org_feature_name;

For the first time run, it works. However, on the second time run, it failed to replace the view with this log. Am I missing something?

19:24:08  Running with dbt=1.0.3
19:24:08  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 165 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
19:24:08  
19:24:14  Concurrency: 1 threads (target='dev')
19:24:14  
19:24:14  1 of 1 START view model ft_analytics_dev.org_feature_name....................... [RUN]
19:24:14  1 of 1 ERROR creating view model ft_analytics_dev.org_feature_name.............. [ERROR in 0.02s]
19:24:15  
19:24:15  Finished running 1 view model in 6.52s.
19:24:15  
19:24:15  Completed with 1 error and 0 warnings:
19:24:15  
19:24:15  Compilation Error in model org_feature_name (models/exp/org_feature_name.sql)
19:24:15    When searching for a relation, dbt found an approximate match. Instead of guessing 
19:24:15    which relation to use, dbt will move on. Please delete ft_analytics_dev.org_feature_name, or rename it to be less ambiguous.
19:24:15    Searched for: ft_analytics_dev.org_feature_name
19:24:15    Found: ft_analytics_dev.org_feature_name
19:24:15    
19:24:15    > in macro create_or_replace_view (macros/materializations/models/view/create_or_replace_view.sql)
19:24:15    > called by macro materialization_view_athena (macros/materializations/models/view/view.sql)
19:24:15    > called by model org_feature_name (models/exp/org_feature_name.sql)
19:24:15  
19:24:15  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
Gatsby-Lee commented 2 years ago

For anyone who got this similar issue.

Simply, dbt doesn't work well with upper case and lower case stuff. I read about this and checked my configuration in schema name ( or DB ) in Athena ( Glue Catalog ) and Table Name. Everything I have were lower case.

So, I was stuck again LOL.

Finally, I found out the the camel case issue caused from Glue DataSource. ( after debugging the dbt-core )

If you use AWS Glue Catalog and it is camel case, use lower case in your dbt profile. I know it sounds weird, but yep if you use all lower case in your profile for the AWS Glue Data Source, then it will work.

I hope this help you,

JavierMonton commented 1 year ago

We had a similar issue, we were testing the connection on the default AwsDataCatalog and we had to configure it as awsdatacatalog in the dbt profile to fix it.