LewisDavies / upstream-prod

A dbt package for easily using production data in a development environment.
32 stars 4 forks source link

Ephemeral models are not supported #12

Closed alittlesliceoftom closed 1 year ago

alittlesliceoftom commented 1 year ago

So if you add this to a project and do a full run, when it hits a model with an upstream ephemeral model, it tries to reference it, instead of inserting it into the code as a CTE like the normal adapter would

alittlesliceoftom commented 1 year ago

Update. The behaviour is slightly more subtle: Context: Model1 is ephemeral model Model2 is a table model depending on model 1

Situation 1 - Fails: dbt run -s Model2

Situation 2 - works: dbt run -s 1+Model2

So where the models are included in scope behaviour is as expected, but where ephemeral models are excluded from scope, they are incorrectly composed. I tried to look into this more, but was struggling to find where dbt main package enacts references. Situation 1 and 2 both work when package is disabled.

LewisDavies commented 1 year ago

Thanks @alittlesliceoftom, I don't use ephemeral models so I've never encountered this.

I fixed the problem by making the package use dbt's built-in ref function for ephemeral models. The tests I added to validate the fix look good but it would be great if you could check against your project too. You can install the new version by adding this to packages.yml:

packages:
  - git: https://github.com/LewisDavies/upstream-prod.git
    revision: ephem-fix
alittlesliceoftom commented 1 year ago

Hey @LewisDavies , Thanks so much for raising the fix, I have confirmed that ephemeral models work right on my env from this revisiion!