[Bug] Setting schema on dbt_project.yml appends instead of overwriting property

[joaocrebelo]

Is this a new bug in dbt-athena?

• [X] I believe this is a new bug in dbt-athena
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The schema property is mandatory when configuring a profile via profile.yml. This property, however, should be overwritable by the models definition in dbt_project.yml, which would allow targeting different Glue databases (aka "schemas") in the same project.

However, at the moment, setting the schema property in profile.yml and dbt_project.yml is appending both and generating a new database which is not supposed to exist.

Here's an example of this behavior that I tested, assuming a model named my_model inside a folder named my_folder

Consider the following snippet of profiles.yml

athena:
  target: dev
  outputs:
    dev:
      type: athena
      schema: default
[...]

And the following snippet of dbt_profiles.yml

[...]
models:
  my_project:
    my_folder:
      +materialized: view
      +schema: custom_schema

Produces a model under: default_custom_schema.my_model

Additional Tests

• Configuring the schema property under dbt_profiles.yml without the + sign also produces the same result
• Configuring the schema property under profiles.yml with an empty string produces a database starting with an underscore like _custom_schema.my_model

Expected Behavior

As with other adapters, overriding the configuration of the schema property in dbt_project.yml for a specific group of models should apply to those models and fully override the default setting from profiles.yml

In the example above, the model should be created under custom_schema.my_model

Steps To Reproduce

1. Setup dbt-core with dbt-athena-community
2. Create a valid configuration for config.yml with a valid schema, like default
3. Create a valid configuration for dbt_profile.yml with a schema under the models key different from the above
4. Run dbt run, see cli output and created objects for reported issue.

Environment

- OS:Ubuntu 22.04
- Python: 3.10
- dbt: 1.8.2
- dbt-athena-community: 1.8.2

Additional Context

No response

[nicor88]

What you raised it's not a bug.

You need to overwrite the macro generate_schema_name to have the behavior that you expect, and it's the same in other adapters. dbt-athena doesn't have the knowledge of overwriting the schemas, this behaviour is implemented in dbt-core.

If you look the original implementation:

{% macro generate_schema_name(custom_schema_name, node) -%}

    {%- set default_schema = target.schema -%}
    {%- if custom_schema_name is none -%}

        {{ default_schema }}

    {%- else -%}

        {{ default_schema }}_{{ custom_schema_name | trim }}

    {%- endif -%}

{%- endmacro %}

you can understsand why default_custom_schema.my_model is produced in your case. Pretty much you have a custom schema in your project configuration, that is concatenated with your default schema in the global profiles. Implementing a macro that does what you need it's the recommended approach to tackle this.

[joaocrebelo]

You are totally right.

The example I checked when I faced the issue was already an overwrite of the macro generate_schema_name that avoided the concatenation and I wrongly assumed it was the original implementation 🤦‍♂️

Thanks for your help and the fast response 👍