dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
286 stars 115 forks source link

Snowflake: When creating external tables, add an option to use database and schema #293

Closed samarpatel-ce closed 2 months ago

samarpatel-ce commented 2 months ago

Describe the feature

Currently snowflake uses database and schema from the profile when it connects to the snowflake. This prevents creation of external tables in a different database/schema than the default database/schema that snowflake connects to.

Describe alternatives you've considered

Manage multiple profiles for each database/schema neeed

Additional context

This is specific to snowflake

Who will this benefit?

Useful for creating external tables in multiple/different schemas at the same time

dataders commented 2 months ago

This is already possible, though I think I can understand your confusion. Ultimately, this package is just an extension of Sources. So this package (under-documented as it is) makes no mention of this feature.

Instead it's described as a feature of Sources (see Add Sources to your DAG: Declaring a source for the example included below)

version: 2

sources:
  - name: jaffle_external
    description: external stage for things
    database: my_other_db  
    schema: some_schema  
    tables:
      - name: order
        external:
         BLAH

The constraint here is that you must use modify each combination of database & schema as a distinct source. In my opinion, that's a good thing, as it forces you to separate your concerns.

I'm closing for now, but feel free to open back up, if I'm mistaken here.