dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
858 stars 166 forks source link

`inline` models do not seem to respect referenced dependencies. #1228

Closed igofunke closed 3 years ago

igofunke commented 3 years ago

Problem

inline models do not seem to respect referenced dependencies.

How to reproduce it

  1. Create a file a.sqlx inside definitions and place the following code.
config {type: "inline"}
select "a" as A
  1. Create a file c.sqlx inside definitions with the following code.
config {type: "inline"}
select "c" AS C
  1. Create a file b.sqlx inside definitions with the following definition.
config {type: "view"}
select * from ${ ref("c") }
  1. Run dataform run --actions b. Note the following error.

    > select * from ()
    bigquery error: Syntax error: Unexpected ")" at [3:16]
  2. Change b.sqlx to read from a.sqlx

    select * from ${ ref("a") }
  3. Run dataform run --actions b. Note the view is created successfully.

Dataset created:  warehouse.b [view]

Expected behaviour

Step 4. should resolve the dependency and bring in the contents of c.sqlx

BenBirt commented 3 years ago

Thanks for the report!

This is a known issue. inline models should not be used any longer (at least in part for this reason), they're not documented (though if you have found somewhere, please tell me and I'll remove it!), and the support only remains in the framework for backwards compatibility with (very) old users.

igofunke commented 3 years ago

Thanks for the fast answer, @BenBirt! I found it digging in the code. I think they are really useful so it's a pity that they have been deprecated. What is the workaround if I want to get an intermediate model that I don't want to materialise in my warehouse? Splitting in intermediate models not only helps reducing complexity but also helps to test multiple parts better.