MaterializeInc / materialize

The data warehouse for operational workloads.
https://materialize.com
Other
5.68k stars 459 forks source link

`REFRESH` introspection #24051

Open hlburak opened 7 months ago

hlburak commented 7 months ago

Users should be able to answer the following questions from the system catalog:

See design doc for discussion and latest plan.

ggevay commented 4 months ago

Recording some recent, specific requirements arising from actual usage of the feature:

ggevay commented 3 months ago

We should probably show the SCHEDULE in mz_clusters.

ggevay commented 3 months ago

We were also thinking about a refresh history that records rehydration times for the last several refreshes, but this is probably not so urgent. Note that this would be somewhat harder to implement then just showing the last and next refresh times.

Edit: The design doc also mentions an mz_materialized_view_refresh_history table. I think we should eventually add such a separate table, because at some point we'll want to record how long the last few rehydrations took, so that the user (or our own heuristics) can tune the warmup period. But for now, probably mz_audit_events will be fine for keeping the history of cluster turn on/offs. We could add a reason into the details column, which is already a json blob, to show which MV prompted a turning on.

ggevay commented 3 months ago

Note that mz_audit_events is going to show when we turn clusters on/off automatically. But maybe we'll want to add some details on the exact reason for turning a cluster on/off, e.g., which MVs necessitated it (see here).

ggevay commented 3 months ago

What is the status of the latest refresh for my view?

The user can see this by running EXPLAIN TIMESTAMP in STRICT SERIALIZABLE mode, and looking at can respond immediately. If it's false, then the last refresh's completion is overdue.

ggevay commented 3 months ago

Eventually, we'll probably also want a next_scheduled_turn_on for clusters, but this is not so urgent. It will get more important when we'll be choosing the warmup time automatically.

ggevay commented 3 months ago

Updated the design doc.