dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
288 stars 116 forks source link

Same external location root with dynamic table/folder name #246

Closed MarkrJames closed 3 days ago

MarkrJames commented 6 months ago

Describe the feature

I believe when listing external sources for these to be registered using dbt run-operation stage_external_sources the external location needs to be specified for each individual table/source name like so:

version: 2
sources:
  - name: datalake
    tables:                         
      - name: table_a
         external:
           location: '/mnt/container/folder/table_a'
           using: parquet
      - name: table_b
         external:
           location: '/mnt/container/folder/table_c'
           using: parquet
      - name: table_c
         external:
           location: '/mnt/container/folder/table_c'
           using: parquet

I'd like to be able to specify location and using once and all table names under that section to use the same location and have a dynamic {name}, something like so:

version: 2
sources:
  - name: datalake
     external:
       location: '/mnt/container/folder/{name}'
       using: parquet
    tables:                         
      - name: table_a
      - name: table_b
      - name: table_c

Additional context

A similar feature can be found in the dbt-duckdb adapter: https://github.com/duckdb/dbt-duckdb?tab=readme-ov-file#reading-from-external-files

Who will this benefit?

github-actions[bot] commented 1 week ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 3 days ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.