search

Teradata / dbt-teradata-utils

Teradata package that provides compatibility for dbt-utils
Apache License 2.0
3 stars 4 forks source link

insert_by_period materializations #11

Closed AdamovichAleksey closed 1 month ago

AdamovichAleksey commented 3 months ago

Describe the feature

The feature is already mentioned in teradata_utils, but marked as "no plans to implement, please submit an issue if you need it"

Describe alternatives you've considered

Not alternative at this moment, I tried to adopt the general solution, but didn't managed.

Who will this benefit?

This feature is extremely necessary to overcome spool issue for our huge datasets.

Are you interested in contributing this feature?

I tried to solve it, but didn't managed.

septimit commented 3 months ago

Hi Aleksey, insert_by_period materialization is now part of dbt-labs-experimental-features repo. I have forked the repo and had a first go at tailoring insert_by_period materialization implementation for Teradata in this branch

Here are some known limitations:

  1. start_date and stop_date must to be specified as timestamps see this sample model
  2. tmp tables (period0_tmp, period1_tmp ...) are not removed after the run is completed and can lead to failures on reruns
  3. period granularity is limited to INTERVAL arithmetic
  4. this list is not complete, since very little testing has been done on the code

I hope this code is useful and can unblock you, Septi