[Bug] DBT docs erroneously complains about duplicate definition of resources

pvanderlinden commented 5 months ago

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt docs generate gives the following error:

Compilation Error
  dbt found two schema.yml entries for the same resource named model.name. Resources and their associated columns may only be described a single time. To fix this, remove one of the resource entries for model.name in this file:
   - models/xxx/properties.yml

while it is only defined once

Expected Behavior

generate the docs

Steps To Reproduce

create a model sql file
create a properties.yml file and add descriptions and column descriptions
generate the docs -> might work
generate the docs -> probably gives the error, it seems to happen mostly when you generate the docs twice
dbt clean
generate the docs -> should work

Relevant log output

Example without modifying any files:

$ dbt docs generate                                                                                                                           
15:45:23  Running with dbt=1.7.11
15:45:23  Registered adapter: snowflake=1.7.3
15:45:23  Encountered an error:
Compilation Error
  dbt found two schema.yml entries for the same resource named model.name. Resources and their associated columns may only be described a single time. To fix this, remove one of the resource entries for model.name in this file:
   - models/some_models/properties.yml

$ dbt clean                                                                                                                                 
15:45:31  Running with dbt=1.7.11
15:45:31  Checking /home/paul/projects/dbt/dbt_packages/*
15:45:31  Cleaned /home/paul/projects/dbt/dbt_packages/*
15:45:31  Checking /home/paul/projects/dbt/target/*
15:45:31  Cleaned /home/paul/projects/dbt/target/*
15:45:31  Finished cleaning all paths.

$ dbt docs generate                                                                                                                        INT ✘  ▼  16:45:34 
15:45:37  Running with dbt=1.7.11
15:45:37  Registered adapter: snowflake=1.7.3
15:45:37  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:45:38  Found 115 models, 32 seeds, 3 operations, 2 tests, 0 sources, 0 exposures, 0 metrics, 449 macros, 0 groups, 0 semantic models
15:45:38
15:45:45  Concurrency: 2 threads (target='dev')
15:45:45
15:45:49  Building catalog
15:45:54  Catalog written to /home/paul/projects/dbt/target/catalog.json

Environment

- OS: Arch Linux
- Python: 3.11.6
- dbt: 1.7.11

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

dbeatty10 commented 5 months ago

Thanks for raising this issue @pvanderlinden !

This looks similar to https://github.com/dbt-labs/dbt-core/issues/4233, but has a distinct error message.

Two questions:

Does it work for you if you use the --no-partial-parse flag (like dbt docs generate --no-partial-parse) ? It should give the same benefit as dbt clean, but a little more ergonomically.
I wasn't able to reproduce this. Can you try to find a simple set of files (and workflow steps) that will reliably reproduce the issue you are seeing?

pvanderlinden commented 5 months ago

Thanks @dbeatty10 --no-partial-parse also works around the issue. I just figured out how to reproduce the issue.

Reproduce steps:

start a project with dbt init
rename the file my_first_dbt_model.sql to test.my_first_dbt_model.sql and add a second column col2
open the schema.yaml file, adjust the name of the model and add the column, also adjust the ref in my_second_dbt_model.sql

add custom_naming_macros.sql to macros with the following content

{% macro generate_alias_name(custom_alias_name=none, node=none) -%}
{%- if custom_alias_name is none -%}
    {%- set split_name = node.name.split('.') -%}
    {%- if split_name|length > 1 -%}
        {{ split_name[-1] | trim }}
    {%- else -%}
        {{ node.name }}
    {%- endif -%}
{%- else -%}
    {{ custom_alias_name | trim }}
{%- endif -%}
{%- endmacro %}

now run dbt docs generate
comment one of the columns in the schema.yaml and save
run dbt docs generate: reproduces the bug

We have this macro in place to have tables/views in different schemas with the same name, e.g.

db.schema1.table_one
db.schema2.table_one

As we can't have two files named table_one.sql even in a different directory, we have to prefix the files (which is included in the "ref").

dbeatty10 commented 5 months ago

Thanks for sharing these details @pvanderlinden 🏆

I was able to reproduce what you are seeing using dbt-duckdb. See below for the smallest reproducible example I could find.

The most impactful piece was:

adding a period within the file name of the downstream model (second) that referenced the upstream model (first).

Details

the override of generate_alias_name was not needed to reproduce
adding a period within the file name for my_first_dbt_model was not needed to reproduce
adding a period within the file name for my_second_dbt_model was needed to reproduce
changing a description field within models/_models.yml in between dbt docs generate was needed to reproduce

Reprex

Assuming you already have a simple dbt_project.yml with a dbt profile configured, here's all that is needed to reproduce:

dbt clean
mkdir -p models
cat << EOF > models/my_first_dbt_model.sql
select 1 as id
EOF
cat << EOF > models/test.my_second_dbt_model.sql
-- {{ ref("my_first_dbt_model") }}
select 2 as id
EOF
cat << EOF > models/_models.yml
models:
  - name: "my_first_dbt_model"
    description: My favorite color is blue
  - name: "test.my_second_dbt_model"
    description: My favorite color is green
EOF
dbt docs generate
cat << EOF > models/_models.yml
models:
  - name: "my_first_dbt_model"
    description: My favorite color is blue
  - name: "test.my_second_dbt_model"
    description: My favorite color is (not actually) green
EOF
dbt docs generate

pvanderlinden commented 5 months ago

A few extra observations on this issue:

the issue also affects dbt run for those models
To work around the issue, we replaced some filenames from using a . to using a -. This doesn't trigger the issue
not directly the same issue: if you go to the table in the docs, the name is correct in the tree on the left, but the title of the page still includes the stripped of prefix.

dbeatty10 commented 5 months ago

These are great observations!

not directly the same issue: if you go to the table in the docs, the name is correct in the tree on the left, but the title of the page still includes the stripped of prefix.

Yep, this looks separate. If you think this is an issue, feel free to open a bug report in the dbt-docs repo, and we can take a closer look.

dbt-labs / dbt-core