dbt-labs / dbt-codegen

Macros that generate dbt code
https://hub.getdbt.com/dbt-labs/codegen/latest/
Apache License 2.0
459 stars 99 forks source link

`generate_base_model()` doesn't work across databases on Redshift #167

Open jeremyyeo opened 5 months ago

jeremyyeo commented 5 months ago

Describe the bug

If your redshift connection is to the database dev but your source is on a separate database sources - then the generate_base_model() macro does not work due to https://github.com/dbt-labs/dbt-redshift/issues/736

Steps to reproduce

  1. Connect your redshift project to a database dev.
  2. Specify a source that is on a separate database:
    # models/sources.yml
    version: 2
    sources:
    - name: my_sources
    database: sources_only
    schema: public
    tables:
      - name: source_foo
  3. Use the codegen.generate_base_model() macro on that source and see that we are not able to list the columns since https://github.com/dbt-labs/dbt-redshift/issues/736 meant that adapter.get_columns_in_relation() does not work across databases for Redshift. (Here is where this package uses get_columns_in_relation())
$ dbt debug
...
02:14:25  Connection:
...
02:14:25    user: root
02:14:25    port: 5439
02:14:25    database: dev
...
$ dbt run-operation generate_base_model --args '{"source_name": "my_sources", "table_name": "source_foo"}'
02:15:00  Running with dbt=1.7.9
02:15:01  Registered adapter: redshift=1.7.1
02:15:03  Found 1 model, 1 source, 0 exposures, 0 metrics, 594 macros, 0 groups, 0 semantic models

with source as (

    select * from {{ source('my_sources', 'source_foo') }}

),

renamed as (

    select

    from source

)

select * from renamed

Expected results

We should be able to get column information across databases and write them into the renamed cte.

Actual results

See above.

Screenshots and log output

See above.

System information

The contents of your packages.yml file:

Which database are you using dbt with?

The output of dbt --version:

Core:
  - installed: 1.7.9 
  - latest:    1.7.10 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.7.3 - Update available!
  - bigquery:   1.7.3 - Update available!
  - snowflake:  1.7.2 - Up to date!
  - redshift:   1.7.1 - Update available!
  - postgres:   1.7.9 - Update available!
  - spark:      1.7.1 - Up to date!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

The operating system you're using:

macOS

**The output of python --version:

Python 3.11.2

Additional context

This won't be fixed in this package until some point we actually fix it in the redshift adapter itself in https://github.com/dbt-labs/dbt-redshift/issues/736

Are you interested in contributing the fix?