dbt-labs / dbt-redshift

dbt-redshift contains all of the code enabling dbt to work with Amazon Redshift
https://getdbt.com
Apache License 2.0
97 stars 59 forks source link

[Column-level encoding] Add support for column-level encoding to Redshift adapter #896

Open gordonr opened 2 months ago

gordonr commented 2 months ago

Is this your first time submitting a feature request?

Describe the feature

Currently the Redshift adapter does not support column-level encoding, which is used to configure compression. This appears to be supported in the Redshift CREATE TABLE statement, as documented here. An approach might be to translate a Redshift-specific dbt config setting to the appropriate CREATE TABLE statement modifications to achieve the desired outomce.

Describe alternatives you've considered

If a dbt model is an incremental view, it would be possible to manually modify column-level encoding after the initial build of the Redshift table, but those changes would be lost if a full refresh were done.

Who will this benefit?

This would theoretically benefit all Redshift customers.

Are you interested in contributing this feature?

No response

Anything else?

No response

TuringND commented 1 day ago

For what is worth, one of the values of having column-level encoding relates to incremental models. Currently compression post-hook makes a deep-copy of the model.

This strategy, though sound for table materializations, antagonizes with incremental materializations as it derives in a full copy of the data. When the model data is big enough, using incremental provides a substantial increase in performance, performance that is lost due to the time spent in the deep-copy.

I was going to post about this in a separate git issue / create a topic in discourse but this feels more appropriate.