Support exclude / tags for the creation of external tables

dbt-labs / dbt-external-tables

dbt macros to stage external sources

https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/

Apache License 2.0

297 stars 119 forks source link

Support exclude / tags for the creation of external tables #129

Closed adrianbartnik closed 9 months ago

adrianbartnik commented 2 years ago

Describe the feature

dbt supports selecting models by tags, e.g. dbt run --select tag:my_tag. We'd like to have a similar functionality for the dbt-external-tables package. The dbt source freshness command already supports this, see here, which is referenced by the README.md.

Describe alternatives you've considered

The alternative would be to list all tables manually using the select argument.

Additional context

We are using Snowflake, but this feature is applicate to all databases.

We are not sure how much complexity this would add the project and other input would be appreciated.

Who will this benefit?

This feature would allow more granular creation of external tables, e.g. in bigger projects if external tables need to be created in different databases by different technical users that create those tables. Using tags or exclude would simplify the code.

jtcohen6 commented 2 years ago

@adrianbartnik I totally hear you, and strongly agree! For better and for worse, this ends up being the same ask as the one in https://github.com/dbt-labs/dbt-external-tables/issues/109.

In order to support node selection, stage_external_sources would need to "graduate" from a macro (executed via run-operation) to an honest-to-goodness dbt task. As laid out in https://github.com/dbt-labs/dbt-external-tables/issues/109#issuecomment-995773358 (options 2+3), we could go one of two ways here:

Short term, medium difficulty: Migrate the logic from this package into dbt-core, and a built-in task (e.g. dbt source external)
Long term, high difficulty: Develop a pluggable task interface in dbt-core, which packages/users could tap into to define their own custom tasks that leverage the built-in capabilities of ManifestTask/GraphRunnableTask

moseleyi commented 1 year ago

It would be great to be able to exclude certain tags. I use external tables from static CSV files (grographical data) that doesn't need to be refreshed until I need to do it manually.. This would be a great addition

github-actions[bot] commented 9 months ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 9 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.