ClickHouse / dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Apache License 2.0
245 stars 100 forks source link

Adding PROJECTION to models #319

Closed hohmannr closed 1 month ago

hohmannr commented 2 months ago

Is your feature request related to a problem? Please describe. It would be awesome if we could add projections to models, this is very helpful when using a semantic layer on top of the data marts, to facilitate and speed up querying for certain filters.

Describe the solution you'd like IMHO, the ideal solution would be to configure the projection as a config parameter (not sure though if dbt supports this):

-- models/marts/touchpoints;
{{
    config(
        materialized='materialized_view',
        engine='MergeTree()',
        order_by='workspace_id, toStartOfHour(updated_at), touchpoint_id',
        projections=[
            { "name": "some_projection", "select": ["touchpoint_id", "some_other_col"], "order_by": ["touchpoint_id"] }
        ]
    )
}}
select * from {{ source('other', 'touchpoints') }}

Describe alternatives you've considered The alternative that we use currently is using a migration script to put the projection on various models, after they have been built, which is not really nice, since the code creating a projection and the model are separate from each other and we need to take care that the database migrations are always up to date with the dbt models.

stephen-up commented 2 months ago

FWIW. We are adding projections using post_hooks.

It works alright, But we get timeouts for materializing slow projections. ClickHouse exception: The read operation timed out

image
hohmannr commented 1 month ago

@BentsiLeviav Thanks for closing this so fast!