dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.68k stars 1.61k forks source link

[Bug] Cannot run unit test with duckdb as database #10292

Closed sogunsemi closed 3 months ago

sogunsemi commented 3 months ago

Is this a new bug in dbt-core?

Current Behavior

When I run the unit tests:

dbt test --select test_type:unit

I get back this error:

Compilation Error in model model_a (models/source_a/sources.yml)
Not able to get columns for unit test 'model_a' from relation "dev"."source_a"."model_a"

> in macro get_fixture_sql (macros/unit_test_sql/get_fixture_sql.sql)
> called by model model_a (models/source_a/source.yml)

Unit test:

unit_tests:
  - name: test_type_conversions
    description: "Check type conversion logic"
    model: model_a
    given:
      - input: source('source_a', 'model_a')
        rows:
          - {year: '2015', value: '2.9'}
    expect:
      rows:
      - {year: 2015, absence_rate_total: 2.9}

Expected Behavior

The unit test runs to completion.

Steps To Reproduce

  1. Configure duckdb as the warehouse using the dbt-duckdb adapter version 1.8.1
  2. Create a unit test for a model
  3. Run it using the command: dbt test --select test_type:unit

Relevant log output

No response

Environment

- OS: MacOS Monterey 12.4
- Python: 3.11.9
- dbt: 1.8.2

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

Using dbt-duckdb 1.8.1

dbeatty10 commented 3 months ago

Thanks for reaching out @sogunsemi !

Does database table/view for that source exist in your DuckDB database?

i.e., Does the following command give a result without error?

dbt show --inline 'select * from {{ source("source_a", "model_a") }}'
sogunsemi commented 3 months ago

Thanks for reaching out @sogunsemi !

Does database table/view for that source exist in your DuckDB database?

i.e., Does the following command give a result without error?

dbt show --inline 'select * from {{ source("source_a", "model_a") }}'

Ah thanks! I get a 403 error which is expected. The data is in a GCS bucket that I don't have local access to.

I thought I would be able to run the unit_tests without hitting the actual data?

dbeatty10 commented 3 months ago

dbt needs to access the metadata of the given inputs, specifically the column names and their data types.

Since the error message you got is expected when you don't have access to the data, I'm going to close this as not a bug.