search

dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
294 stars 119 forks source link

Enable an errored creation of external tables to not stop following ones #174

Closed dgreen161 closed 1 year ago

dgreen161 commented 1 year ago

Describe the feature

A clear and concise description of what you want to happen.

Currently if an external table was unable to be created, it stops the following ones from being created. If there was a way to either handle the error or create them in parallel, this would help alleviate the issue.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Running it from dbt Cloud, we considered creating multiple steps in the job but since a failed step stops the following ones, this won't work.

Additional context

Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here.

I think this would be database agnostic. In this case, the issue was found when mounting from GCS to BigQuery.

Who will this benefit?

What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly.

Anyone with multiple external tables being created at the same time.

Jay-code0 commented 1 year ago

+1 of importance to this feature, currently if you have multiple external sources and run through them, if 1 fails the rest wont be loaded, a skip error config would be really useful as it currently blocks loading of all other tables when an error occurs for 1

github-actions[bot] commented 1 year ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 1 year ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.