Set clustering on query history and warehouse_events

sundeck-io / OpsCenter

OpsCenter for Snowflake makes it easy to understand and manage your Snowflake consumption

Other

22 stars 15 forks source link

Set clustering on query history and warehouse_events #650

Closed joshelser closed 4 months ago

joshelser commented 4 months ago

Uses a row in catalog.config to denote when clustering has been enabled already, and avoids setting the cluster by multiple times. We don't have any ability to remove rows from catalog.config (per my memory and a cursory glance), so I think that's safe.

Tested both the upgrade and fresh install case. I'm double-checking that the block is correctly run exactly once by hand.

jacques-n commented 4 months ago

Can you also add a new column in the downstream views of ST_PERIOD::date as ST_DAY.

Let's also add the same kinds of clustering to cluster_sessions/warehouse_sessions.

joshelser commented 4 months ago

Can you also add a new column in the downstream views of ST_PERIOD::date as ST_DAY.

Went through every view in reporting (that isn't coming from simple_data_events_maintenance) and made sure it has a ST_DAY column.

Let's also add the same kinds of clustering to cluster_sessions/warehouse_sessions.

Clustering on (RECORD_TYPE, ST_PERIOD::DATE), exactly once, also added to INTERNAL_REPORTING_MV.CLUSTER_AND_WAREHOUSE_SESSIONS_COMPLETE_AND_DAILY

joshelser commented 4 months ago

unit-test failure should be fixed by #648. Will rebase this on top of that commit before merging here.

joshelser commented 4 months ago

Rebased on #648. Will also merge this one if CI is good without any more product-changes.