dbt-labs / dbt-rpc

A server that can compile and run queries in the context of a dbt project. Additionally, it provides methods that can be used to list and terminate running processes.
https://docs.getdbt.com/reference/commands/rpc
Apache License 2.0
19 stars 7 forks source link

[CT-2660] [CT-2648] [Regression] Since 1.5 `--vars` is not rendered in the model config header #145

Open b-per opened 1 year ago

b-per commented 1 year ago

Is this a regression in a recent version of dbt-core?

Current Behavior

In 1.5, running dbt run -s var_mat --vars "materialized: table" on the following model creates it as a view.

{% set var_mater = var('materialized','view') %}

{{
    config(
        materialized=var_mater
    )
}}

select 1 as id
-- materialized: {{ model.config.materialized }}
-- config: {{ var_mater }}

This is confirmed with the logs showing

select 1 as id
-- materialized: view
-- config: table

Expected/Previous Behavior

Until 1.4 the model is correctly created as a table. I would expect to still be the case with 1.5 onwards.

Steps To Reproduce

Relevant log output

No response

Environment

- OS: tested on dbt Cloud
- dbt (working version): 1.4
- dbt (regression version): 1.5

Which database adapter are you using with dbt?

No response

Additional Context

No response

jtcohen6 commented 1 year ago

@b-per I'm not able to reproduce this -

-- models/var_mat.sql
{% set var_mater = var('materialized','view') %}

{{
    config(
        materialized=var_mater
    )
}}

select 1 as id
-- materialized: {{ model.config.materialized }}
-- config: {{ var_mater }}
$ dbt run -s var_mat --vars "materialized: table"
13:28:59  Running with dbt=1.5.1
13:28:59  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 308 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
13:28:59
13:28:59  Concurrency: 5 threads (target='dev')
13:28:59
13:28:59  1 of 1 START sql table model dbt_jcohen.var_mat ................................ [RUN]
13:28:59  1 of 1 OK created sql table model dbt_jcohen.var_mat ........................... [SELECT 1 in 0.12s]
13:28:59
13:28:59  Finished running 1 table model in 0 hours 0 minutes and 0.49 seconds (0.49s).
13:28:59
13:28:59  Completed successfully
13:28:59
13:28:59  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Compiled code:

select 1 as id
-- materialized: table
-- config: table
b-per commented 1 year ago

This is very odd. The exact same model and command doesn't return the expected results on dbt Cloud on 1.5.1. It was actually raised to me by a user who also sees the same behaviour as I see.

I will try to investigate a bit more

b-per commented 1 year ago

I just ran a job with the exact same config and it created it as a table. This is a strong smell that this is a dbt-rpc bug, then, no?

jtcohen6 commented 1 year ago

This is a strong smell that this is a dbt-rpc bug, then, no?

Could be...

Update: I was able to reproduce this on the IDE using the legacy backend (dbt-rpc). My guess would be that we are no longer triggering a full re-parse in cases where the --vars being passed into the cli_args method are different from the ones saved during the previous parse.

If this is "just fixed" when we move to the newer backend, that would inform the priority of solving this IMO. Let's make sure that's the case once it's available for alpha/beta testing. I'm going to move this to the dbt-rpc repo in the meantime.

will-sargent-dbtlabs commented 1 year ago

@jtcohen6 We have a customer in my cohort, that is experiencing this behaviour, but has it been verified that 1.6 dbt-server will likely solve this? Should I ask them to test the 1.6 (patch prerelease) that is currently available to them, or should they wait for the 1.6 stable?

jeremyyeo commented 1 year ago

@jtcohen6 We have a customer in my cohort, that is experiencing this behaviour, but has it been verified that 1.6 dbt-server will likely solve this? Should I ask them to test the 1.6 (patch prerelease) that is currently available to them, or should they wait for the 1.6 stable?

image

Fwiw - current iteration using 1.6 (so not dbt-server) shows this is not fixed yet.

jtcohen6 commented 1 year ago

Thanks @will-sargent-dbtlabs @jeremyyeo! This is being tracked here: https://dbtlabs.atlassian.net/browse/XP-2506

Note that there will be a performance penalty if you are overriding --vars on the CLI (just as there already is for dbt-core) because it requires a full re-parse of the project today.

Thinking through ways we can make that better over here: