dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
405 stars 227 forks source link

Update adaptors.sql to use `tblproperties` macro #848

Closed etheleon closed 1 year ago

etheleon commented 1 year ago

resolves #865 resolves #190 docs dbt-labs/docs.getdbt.com/#

Problem

tblproperties clause macro is not used when creating tables. Add macro to adaptor.sql

Solution

Add macro

Checklist

etheleon commented 1 year ago

see discussion https://github.com/dbt-labs/dbt-spark/discussions/847

JCZuurmond commented 1 year ago

@fleid: Could you merge this PR?

etheleon commented 1 year ago

@colin-rogers-dbt can you review this? :bow:

Fleid commented 1 year ago

@JCZuurmond are we doing this one? Looks like the changelog entry is missing.

colin-rogers-dbt commented 1 year ago

@etheleon can you add a changelog entry?

etheleon commented 1 year ago

hey @colin-rogers-dbt working on it 👍

/*
    Welcome to your first dbt model!
    Did you know that you can also configure models directly within SQL files?
    This will override configurations stated in dbt_project.yml

    Try changing "table" to "view" below
*/

{{
    config(
        location_root='s3a://some_bucket/tables',
        tblproperties = {
          'format': 'iceberg/parquet',
          'format-version': '2',
          'write.delete.mode': 'merge-on-read',
          'write.delete.target-file-size-bytes': '536870912',
          'write.merge.mode': 'merge-on-read',
          'write.metadata.delete-after-commit.enabled': 'true',
          'write.metadata.previous-versions-max': '100',
          'write.target-file-size-bytes': '536870912',
          'write.update.mode': 'merge-on-read'}
    )
}}

with source_data as (

    select 1 as id
    union all
    select null as id

)

select *
from source_data

/*
    Uncomment the line below to remove records with null `id` values
*/

-- where id is not null

output from SHOW CREATE TABLE

CREATE TABLE Iceberg catalog.ds.my_first_dbt_model ( id INT) USING iceberg LOCATION 's3a://some_bucket/tables/my_first_dbt_model' TBLPROPERTIES ( 'current-snapshot-id' = '1344159441507971776', 'format' = 'iceberg/parquet', 'format-version' = '2', 'write.delete.mode' = 'merge-on-read', 'write.delete.target-file-size-bytes' = '536870912', 'write.merge.mode' = 'merge-on-read', 'write.metadata.delete-after-commit.enabled' = 'true', 'write.metadata.previous-versions-max' = '100', 'write.target-file-size-bytes' = '536870912', 'write.update.mode' = 'merge-on-read')