Error in `base_source_columns` and `base_node_columns` in Redshift

stevenconnorg commented 3 months ago

Describe the bug

Errors in base_source_columns and base_node_columns in Redshift:

(dbt-env-1.8) (dbt-env-1.6) stevengonzalez@Steven-Gonzalez dbt-campaign-workflows % dbt build -s  base_node_columns base_source_columns
17:08:11  Running with dbt=1.8.3
17:08:12  [WARNING]: Deprecated functionality
The `tests` config has been renamed to `data_tests`. Please see
https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more
information.
17:08:12  Registered adapter: redshift=1.8.1
17:08:18  Found 734 models, 34 snapshots, 83 seeds, 3215 data tests, 1137 sources, 1 exposure, 1801 macros
17:08:19  
17:08:22  Concurrency: 4 threads (target='dev')
17:08:22  
17:08:22  1 of 2 START sql table model sgonzalez.base_node_columns ....................... [RUN]
17:08:22  2 of 2 START sql table model sgonzalez.base_source_columns ..................... [RUN]
17:08:35  1 of 2 ERROR creating sql table model sgonzalez.base_node_columns .............. [ERROR in 13.89s]
17:08:43  2 of 2 ERROR creating sql table model sgonzalez.base_source_columns ............ [ERROR in 21.60s]
17:08:44  
17:08:44  Finished running 2 table models in 0 hours 0 minutes and 24.93 seconds (24.93s).
17:08:45  
17:08:45  Completed with 2 errors and 0 warnings:
17:08:45  
17:08:45    Database Error in model base_node_columns (models/staging/graph/base/base_node_columns.sql)
value too long for type character varying(1)
compiled Code at target/run/dbt_project_evaluator/models/staging/graph/base/base_node_columns.sql
17:08:45  
17:08:45    Database Error in model base_source_columns (models/staging/graph/base/base_source_columns.sql)
INSERT has more expressions than target columns
compiled Code at target/run/dbt_project_evaluator/models/staging/graph/base/base_source_columns.sql

Steps to reproduce

Running on dbt-core/dbt-redshift v1.8.0 with dbt-project-evaluator 0.12.0

Expected results

Get Models to run and insert data via post-hook

Actual results

N/A

Screenshots and log output

System information

The contents of your packages.yml file:


packages:
  - package: dbt-labs/audit_helper
    version: 0.12.0
  - package: dbt-labs/dbt_utils
    version: 1.2.0
  - package: dbt-labs/codegen
    version: 0.12.1
  - package: calogica/dbt_expectations
    version: 0.10.3
  - package: elementary-data/elementary
    version: 0.15.2
  - package: dbt-labs/redshift
    version: 0.9.0  
  - package: data-mie/dbt_profiler
    version: 0.8.2
  - package: dbt-labs/dbt_project_evaluator
    version: 0.12.0packages:
  - package: dbt-labs/audit_helper
    version: 0.12.0
  - package: dbt-labs/dbt_utils
    version: 1.2.0
  - package: dbt-labs/codegen
    version: 0.12.1
  - package: calogica/dbt_expectations
    version: 0.10.3
  - package: elementary-data/elementary
    version: 0.15.2
  - package: dbt-labs/redshift
    version: 0.9.0  
  - package: data-mie/dbt_profiler
    version: 0.8.2
  - package: dbt-labs/dbt_project_evaluator
    version: 0.12.0

Which database are you using dbt with?

[ ] postgres
[x] redshift
[ ] bigquery
[ ] snowflake
[ ] trino/starburst
[ ] other (specify: ____)

The output of dbt --version:

(dbt-env-1.8) (dbt-env-1.6) stevengonzalez@Steven-Gonzalez dbt-campaign-workflows % dbt --version
Core:
  - installed: 1.8.3
  - latest:    1.8.3 - Up to date!

Plugins:
  - redshift: 1.8.1 - Up to date!
  - postgres: 1.8.2 - Up to date!

Additional context

Are you interested in contributing the fix?

Yes! I'll link a PR to this issue with the fix I've employed locally

nevdelap commented 3 months ago

I believe this issue is the same as what I'm seeing in Snowflake. Update: Ah, I see #465. :+1:

dbt-core: 1.8.3 dbt-snowflake: 1.8.3 Python: 3.11

02:51:34    Database Error in model base_source_columns (models/staging/graph/base/base_source_columns.sql)
  002020 (21S01): SQL compilation error:
  Insert value list does not match column list expecting 5 but got 8
  compiled Code at target/run/dbt_project_evaluator/models/staging/graph/base/base_source_columns.sql

b-per commented 3 months ago

Yes, it looks like #465 .

Let's discuss about it there. As we suggested on the other issue, it would be useful to see what SQL is being sent. Because we load data in hooks though it doesn't show under target/... and we can only see them in the dbt logs or in the Data Warehouse query history.

b-per commented 3 months ago

So the issue with base_source_columns is now tracked in #465 and it is getting fixed.

stevenconnorg commented 1 month ago

Thank you! This issue has been resolved.

dbt-labs / dbt-project-evaluator