[CT-1877] [Bug] Jinja expression is not parsed under tests meta

IDoneShaveIt commented 1 year ago

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When defining a test's meta field with jinja expression, the jinja expression is not being parsed and returned as string "{{ jinja_expression }}".

For example:

  - name: customers
    meta:
      channel: "{{ 'one' if 1 == 2 else 'two' }}"
    description: This table has basic information about a customer, as well as some derived facts based on a customer's orders

    columns:
      - name: customer_id
        description: This is a unique identifier for a customer
        tests:
          - unique:
              meta:
                channel: "{{ 'one' if 1 == 2 else 'two' }}"

When looking at the customers model node, the meta's channel field is set to two. When looking at the unique test node, the meta's channel field is set to "{{ 'one' if 1 == 2 else 'two' }}".

Expected Behavior

Both meta's channel field should be set to two (The jinja should be parsed on both cases)

Steps To Reproduce

Define a field under test's meta with jinja expression (at the schema.yml)
Run dbt command and check the graph

Relevant log output

No response

Environment

- OS: macOS
- Python: Python 3.9.13
- dbt: 1.3.2

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

jtcohen6 commented 1 year ago

Thanks for opening @IDoneShaveIt! I was able to reproduce this, using the newer location of meta as well (nested under config, since v0.21):

version: 2
models:
  - name: customers
    description: "This table has basic information about a customer, as well as some derived facts based on a customer's orders"
    config:
      meta:
        channel: "{{ 'one' if 1 == 2 else 'two' }}"
    columns:
      - name: customer_id
        description: This is a unique identifier for a customer
        tests:
          - unique:
              config:
                meta:
                  channel: "{{ 'one' if 1 == 2 else 'two' }}"

$ dbt ls --output json --output-keys name,meta
{"name": "customers", "meta": {"channel": "two"}}
{"name": "unique_customers_customer_id", "meta": {"channel": "{{ 'one' if 1 == 2 else 'two' }}"}}

I think what's going on here is that meta is a dictionary, and the logic here only renders config values if they're strings: https://github.com/dbt-labs/dbt-core/blob/93619a9a3761d3606d257c1274828ac4eef4f627/core/dbt/parser/generic_test_builders.py#L265-L268

We need to loop through the meta keys and render each value one by one. We could treat meta as a special case, or try to apply logic similar to what we do in deep_map_render / render_data.

jtcohen6 commented 1 year ago

Small housekeeping: reclassifying this from bug to enhancement. We should aim for consistent behavior wherever possible, and users have come to expect "functional Jinja" for lots of conditional config — but it's never worked before in this particular spot (AFAIK), nor is it explicitly documented as being supported here.

leogodin217 commented 1 year ago

I think it is an important enhancement, because meta is the only method of configuring how 3rd-party tools interact with dbt. Without this ability, it will be more difficult for the dbt ecosystem to keep up with other transform solutions. Most of us are far enough along, that we are moving to more metadata-driven workflows and the meta config is a big part of it.

dbt-labs / dbt-core