dbt-labs / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
https://dbt-athena.github.io
Apache License 2.0
228 stars 100 forks source link

[Bug] on_schema_change failure with alternate schema #751

Open magno32 opened 3 weeks ago

magno32 commented 3 weeks ago

Is this a new bug in dbt-athena?

Current Behavior

I have an incremental table int_nightly_sales, that is an intermediate table in our workflow.

When attempting to add a column to this table, I get a failure:

16:17:13  dbt.adapters.athena.constants adapter: Error running SQL: alter table `base_db_intermediate`.`int_nightly_sales`
          replace columns (<columns>)

16:17:13  On model.daily_sales_sync.int_nightly_sales: Close
16:17:13  Runtime Error in model int_nightly_sales (models/intermediate/upload/int_nightly_sales.sql)
  FAILED: SemanticException [Error 10072]: Database does not exist: base_db

The table should actually exist on: base_db_intermediate as called out in the config:

{{ config(
    materialized='incremental',
    incremental_strategy='append',
    partitioned_by=['processed_on'],
    on_schema_change='sync_all_columns',
    schema='intermediate'
) }}

I dug through the adapter source and could not find a culprit on this, but the odd thing is the first message references the correct db, where the semantic error thrown references the base_db (which does not actually contain anything at all).

Expected Behavior

The table exists on the proper database in athena, the adapter should reference the correct database instead of the base database for on_schema_change.

Steps To Reproduce

  1. Create an athena dbt project with a base schema (base)
  2. Create an incremental model with a different schema (sub) -> base_sub
  3. dbt build
  4. Add a column to the model
  5. dbt build

This should fail as described.

Environment

- OS: Mint
- Python: 3.10.12
- dbt: 1.8.4
- dbt-athena-community: 1.8.4

Additional Context

No response