markporoshin / dbt-greenplum

Adaptation postgres adapter for Greenplum
36 stars 22 forks source link

Add possibility to create heap based tables #13

Closed Sonofendor closed 1 year ago

Sonofendor commented 1 year ago

Append Only GreemPlum tables can take up a lot of space on disks that is actually not being utilized. Sometimes it is required to create heap based tables for dbt-models. Desired behavior

{{
    config(
        materialized='table',
        appendonly=false,
        schema='test',
    )
}}

select
    floor(random()*(1000))+10 as id,
    current_timestamp as ts

In order for example above to work and create not Append Only table it should produce following SQL statement:

create  table "test"."test_datamart_tmp"
      with (
            appendonly=False
      )
      as (
select
    floor(random()*(1000))+10 as id,
    current_timestamp as ts
      )
      DISTRIBUTED RANDOMLY
      ;

SQL statement should not cotain any of blocksize, orientation, compresstype, compresslevel parameters otherwise GreenPlum produces ERROR: invalid option "orientation" for base relation Hint: Table orientation only valid for Append Only relations, create an AO relation to use table orientation.

Current behavior Currently example above produces following SQL statement:

create  table "test"."test_datamart__dbt_tmp"
      with (
            appendonly=False,
            blocksize=32768,
            orientation=column,
            compresstype=NONE,
            compresslevel=4
      )
      as (
select
    floor(random()*(1000))+10 as id,
    current_timestamp as ts
      )
      DISTRIBUTED RANDOMLY
      ;

Possible solution: Check for parameter appendonly when generating SQL statements to run and if it is set to false do not include blocksize, orientation, compresstype, compresslevel parameters in SQL statement.

Context: dbt-core 1.2.2 dbt-greenplum 1.2.0

markporoshin commented 1 year ago

WIP

markporoshin commented 1 year ago

Please check the pr I have linked to this issue