Open gordonr opened 3 months ago
For what is worth, one of the values of having column-level encoding relates to incremental models. Currently compression post-hook makes a deep-copy of the model.
This strategy, though sound for table materializations, antagonizes with incremental materializations as it derives in a full copy of the data. When the model data is big enough, using incremental provides a substantial increase in performance, performance that is lost due to the time spent in the deep-copy.
I was going to post about this in a separate git issue / create a topic in discourse but this feels more appropriate.
Is this your first time submitting a feature request?
Describe the feature
Currently the Redshift adapter does not support column-level encoding, which is used to configure compression. This appears to be supported in the Redshift CREATE TABLE statement, as documented here. An approach might be to translate a Redshift-specific dbt config setting to the appropriate CREATE TABLE statement modifications to achieve the desired outomce.
Describe alternatives you've considered
If a dbt model is an incremental view, it would be possible to manually modify column-level encoding after the initial build of the Redshift table, but those changes would be lost if a full refresh were done.
Who will this benefit?
This would theoretically benefit all Redshift customers.
Are you interested in contributing this feature?
No response
Anything else?
No response