dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.85k stars 1.63k forks source link

[Bug] Unit test fails with versioned inputs #10880

Open clairegchen opened 1 week ago

clairegchen commented 1 week ago

Is this a new bug in dbt-core?

Current Behavior

Unit tests fail if an input reference is a versioned model even though it is explicitly defined in both the model & the unit test input. The following is the error when the test fails: Compilation Error in unit_test unit_test_mart__model_b (models/mart/unit_test.yml) Unit_Test 'unit_test.repo.mart__model_b.unit_test_mart__model_b’ (models/mart/unit_test.yml) depends on a node named ‘input_versioned_model’ which was not found

Expected Behavior

Unit test should run with specified versioned model.

Steps To Reproduce

  1. Create a model with at least 1 version and have it saved with _v#, e.g. input_versioned_model_v1

  2. This input_versioned_model_v1 has model.yml configured with latest_version: 1 Code for reference:

    - name: input_versioned_model_v1 
    latest_version: 1
    columns:
      - name: id
        ...
    versions:
      - v: 1
  3. Create mart__model_b that references this input_versioned_model without the versioning Code for reference:

    with iv as (
    select *
    from
        {{ ref('input_versioned_model') }}
    ),
    im as (
    select *
    from
        {{ ref('input_model2') }}
    )
    select 
    iv.id,
    iv.category,
    im.profile_id,
    im.type
    from 
    iv
    join im
    on iv.id = im.customer_id
  4. mart__model_b builds correctly with dbt build -s +mart__model_b

  5. Create a dbt unit test for mart__model_b Code for reference:

    
    - name: unit_test_mart__model_b
    model: mart__model_b
    given:
      - input: ref('input_versioned_model', v='1')
        rows:
        - {id: 123, category: electronics}
      - input: ref('input_model2')
        rows:
        - {customer_id: 123, profile_id: 2356, type: onetime}
    expect:
      rows:
      - {id: 123, category: electronics, profile_id: 2356, type: onetime}
Note: I tried the following versioned inputs:
`input: ref('input_versioned_model', v='1')`
`-input: ref('input_versioned_model', v=1)`
7. Run `dbt test --select "mart__model_b,test_type:unit"` will give you the following error:
`Compilation Error in unit_test unit_test_mart__model_b (models/mart/unit_test.yml)
  Unit_Test 'unit_test.repo.mart__model_b.unit_test_mart__model_b’ (models/mart/unit_test.yml) depends on a node named ‘input_versioned_model’ which was not found`

### Relevant log output

```shell
`Compilation Error in unit_test unit_test_mart__model_b (models/mart/unit_test.yml)
  Unit_Test 'unit_test.repo.mart__model_b.unit_test_mart__model_b’ (models/mart/unit_test.yml) depends on a node named ‘input_versioned_model’ which was not found`

Environment

- OS:
- Python: 3.12.6
- dbt-core: 1.8.3 & tried with 1.8.6 & 1.8.7
Plugins:
  - redshift: 1.8.1

Which database adapter are you using with dbt?

redshift: 1.8.1

Additional Context

Found following github issues that are related: 10528 and 10623 and tried the solutions provided.

devmessias commented 1 week ago

Have you tried this ?

with iv as (
    select *
    from
        {{ ref('input_versioned_model', v=1) }}
),

I found a problem with versioned refs that I was able to solve with #10889