When trying to raise a DbtDatabaseError, dbt logs glue adapter: module 'dbt.exceptions' has no attribute 'DbtDatabaseError' but it is not catch by dbt. It then results in a Succeded model run !
Steps To Reproduce
I can raise the error when trying to --full-refresh a Delta incremental model and changing the type of a column.
First, create a minimal model that (called test_database_error):
{{
config(
materialized='incremental',
incremental_strategy='append',
file_format='delta',
)
}}
select 1 as id
Run it a first time
dbt run -s test_database_error
Then change the same model to :
{{
config(
materialized='incremental',
incremental_strategy='append',
file_format='delta',
)
}}
select "a" as id
Finally, run :
dbt run -s test_database_error --full-refresh
You should get a similar terminal output to the following :
09:20:37 Running with dbt=1.8.4
09:20:38 Registered adapter: glue=1.8.1
09:20:39 Found 41 models, 2 snapshots, 1 seed, 2 operations, 1 test, 1 source, 1367 macros
09:20:39
09:20:40
09:20:40 Running 1 on-run-start hook
09:20:40 Concurrency: 1 threads (target='local')
09:20:40
09:20:40 1 of 1 START sql incremental model xxx.test_database_error ............ [RUN]
09:20:40 inside make_temp_relation about to dispatch
09:20:46 Glue adapter: Glue returned `error` for statement None for code
spark.sql("""
select "a" as id
""").write.format("delta").mode("overwrite").save("s3://xxx/xxx/test_database_error")
SqlWrapper2.execute("""select 1""")
, AnalysisException: Failed to merge fields 'id' and 'id'. Failed to merge incompatible data types IntegerType and StringType
09:20:46 Glue adapter: module 'dbt.exceptions' has no attribute 'DbtDatabaseError'
09:20:51 Glue adapter: decoding to str: need a bytes-like object, NoneType found
09:20:51 Glue adapter: Parameter validation failed:
Missing required parameter in TableInput: "Name"
09:20:56 1 of 1 OK created sql incremental model xxx.test_database_error ....... [OK in 15.42s]
09:21:12
09:21:12
09:21:12 Finished running 1 incremental model, 2 project hooks in 0 hours 0 minutes and 33.29 seconds (33.29s).
09:21:12
09:21:12 Completed successfully
09:21:12
09:21:12 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
Expected behavior
When an error is raised, the run should finish in an error.
Screenshots and log output
See sections above.
System information
The output of dbt --version:
Core:
- installed: 1.8.3
- latest: 1.8.4 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- glue: 1.8.1 - Up to date!
- spark: 1.8.0 - Up to date!
The operating system you're using:MacBook Pro 2,3 GHz Intel Core i9 8 cœurs
The output of python --version:Python 3.12.4
Additional context
We should only get the DbtDatabaseError from dbt_common and not dbt since dbt refactored its code a while ago but this was not modified in the adapter.
Describe the bug
When trying to raise a DbtDatabaseError, dbt logs glue adapter: module 'dbt.exceptions' has no attribute 'DbtDatabaseError' but it is not catch by dbt. It then results in a Succeded model run !
Steps To Reproduce
I can raise the error when trying to --full-refresh a Delta incremental model and changing the type of a column. First, create a minimal model that (called test_database_error):
Run it a first time
Then change the same model to :
Finally, run :
You should get a similar terminal output to the following :
Expected behavior
When an error is raised, the run should finish in an error.
Screenshots and log output
See sections above.
System information
The output of
dbt --version:The operating system you're using:
MacBook Pro 2,3 GHz Intel Core i9 8 cœursThe output of
python --version:Python 3.12.4Additional context
We should only get the DbtDatabaseError from dbt_common and not dbt since dbt refactored its code a while ago but this was not modified in the adapter.