dbt-labs / dbt-bigquery

dbt-bigquery contains all of the code required to make dbt operate on a BigQuery database.
https://github.com/dbt-labs/dbt-bigquery
Apache License 2.0
212 stars 149 forks source link

BigQuery tags do not work #1191

Open ThomasLieb-E4U opened 4 months ago

ThomasLieb-E4U commented 4 months ago

Is this a new bug in dbt-bigquery?

Current Behavior

dbt labels with an empty string for the label value produce a BigQuery label with empty value, but no BigQuery tag.

Screenshot of the BigQuery table details: Screenshot 2024-04-24 115151

Expected Behavior

Per dbt-bigquery documentation I expect the dbt label with empty value to result in a BigQuery tag.

Steps To Reproduce

Model file:

{{
  config(
    materialized = "table",
    labels = {'contains_pii':''}
  )
}}

select 1 as num, "text" as txt

Run model and inspect table details in BigQuery.

Relevant log output

No response

Environment

- OS: Ubuntu 22.04.3
- Python: 3.10.12
- dbt-core: 1.7.13
- dbt-bigquery: 1.7.7

Additional Context

No response

dataders commented 4 months ago

reproduction

I was able to reproduce this by creating a brother_ptouch.sql model like the below.

--brother_ptouch.sql
{{
  config(
    materialized = "table",
    labels = {'contains_pii':''}
  )
}}

SELECT
    "brother" as label_printer_make,
    "p_touch" as label_printer_model

The compiled SQL betrays the error here labels = [('contains_pii', '')] when it should be tags = ['contains_pii']

--compiled SQL
create or replace table `dbt-test-env`.`anderstest`.`brother_ptouch`
    OPTIONS(labels = [('contains_pii', '')]) as (
        SELECT 
            "brother" as label_printer_make,
            "p_touch" as label_printer_model
    );

next steps

but I'm still not sure if this is a bug (never was supported) or a regression (was supported at one time but no longer.

This docs section predates the docs site being hosted on Docusaurus.

BigQuery table and view tags can be created by supplying an empty string for the label value.

The PR that I see touching some of this is #209, but not sure if this is root cause.

solutions

some options

  1. Modify BigQueryAdapter.get_common_options()'s conditional about label config to convert a label to a a tag if the value is an empty string ``.
  2. just accept label as an alternative config instead of extending tag config.

https://github.com/dbt-labs/dbt-bigquery/blob/3bd44ac570d54883bf3a45998f27662bfe10c985/dbt/adapters/bigquery/impl.py#L758-L760

jtcohen6 commented 2 months ago

Per BigQuery docs, these are different kinds of "tags" (emphasis mine):

A label that has a key with an empty value is sometimes called a tag. This should not be confused with a tag resource. For more information, see labels and tags. You can create a new label with no value, or you can remove a value from an existing label key.

Labels without values can be useful in situations where you are labeling a resource, but you do not need the key-value format. For example, if a table contains test data that is used by multiple groups, such as support or development, you can add a test_data label to the table to identify it.

ThomasLieb-E4U commented 2 months ago

@jtcohen6, that's a good point. In the sense of tags as empty labels, the documentation is technically correct, but still ambiguous.

To me the main advantage of being able to set BigQuery tags in dbt, is to use conditional IAM policies. But I'm not 100% sure about the added value given that tags, tag bindings, etc need to be managed separately.

msmahliss commented 1 month ago

I also interpreted the docs to be referring to tag resources as I too would like to attach tags to tables to leverage IAM conditions for table-level access control.

The BigQuery docs that @jtcohen6 surfaced do make it seem that the observed behavior is expected and that the term "tag", as used in the docs, refers to labels with no value. If that is the case, the ability to attach tag resources would be a feature request and this issue is correctly labeled as enhancement. Will likely use a post-hook to get around this in the meantime, but would love to see this feature in the future.

VicenteSep commented 1 month ago

@msmahliss How did you do it with post hook? is there a SQL statement that can apply tags to a table?

msmahliss commented 1 month ago

@VicenteSep

Thought it could be done with an ALTER statement. But apparently tags are not supported as an option in the table options list 😞