search

dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
297 stars 119 forks source link

Equivalent of codegen `generate_source` for external tables #143

Closed amirbtb closed 1 year ago

amirbtb commented 2 years ago

Describe the feature

An equivalent of generate_source for external tables : A macro to generate YAML for a source that is an external table. We could provide a bucket argument (to define the S3/Cloud Storage/... bucket) and an optional prefix to restrict the code generation to a subset of the bucket. We could also use another argument to ignore the sources that are already defined in the YAML file or created as external tables in the database.

Describe alternatives you've considered

I am currently using a (sunday) python script with pandas and pandas-gbq to generate the YAML for my parquet sources. It's working only on BigQuery (although quite slow).

Additional context

This feature is not database-specific and could be used with all adapters/databases.

Who will this benefit?

Anyone who needs to define external tables as sources.

github-actions[bot] commented 1 year ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 1 year ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.