Select modified source models only

dbt-labs / dbt-external-tables

dbt macros to stage external sources

https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/

Apache License 2.0

288 stars 116 forks source link

Select modified source models only #256

Open OGrohmann opened 4 months ago

OGrohmann commented 4 months ago

Describe the feature

I would like to integrate the stage_external_sources macro into our dbt ci job in a way that only new and modified source models are updated, similar to the modified statement in the build command (i.e. dbt build --select state:modified).

Describe alternatives you've considered

Currently we are just refreshing all sources in the ci job as a first step, which is very inefficient of course.

Additional context

This request is referring to the select argument of the stage_external_sources macro.

Who will this benefit?

the execution of our ci job for integration testing will be much faster.

dataders commented 4 months ago

totally get it @OGrohmann! it should just know, right?

I'd be curious to know:

how many external tables you have in your project?
how often new external tables are added?
how often external table definitions change?
How long does it take for all the external sources to be staged?

That said just as @jtcohen6 responded in #263, we plan for external tables to live within Core and have access to state-based selectors like you propose.

Meanwhile, we'd love your eyes on https://github.com/dbt-labs/dbt-adapters/discussions/92, as we think about a real future for external tables

OGrohmann commented 3 months ago

Hi @dataders , sorry for the late response! Let me feedback to your questions:

how many external tables you have in your project? Currently 548 objects (Snowflake external tables & Snowpipes counted together)
how often new external tables are added? Regularly for every new request. Maybe 10-20 per month.
how often external table definitions change? Not frequently.
How long does it take for all the external sources to be staged? Normal run without full refresh takes around 25-30 minutes

As a workaround we thought to exclude the macro from our CI and CD jobs and manually update the affected sources upfront. Problem is that this cannot be done before the changes are merged to the main branch and the CI / CD jobs are immediately triggered after creating / completing the pull request.