search

MaterializeInc / materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.
https://materialize.com
Other
5.71k stars 466 forks source link

Consider implementing operator hydration status tracking as a view #27753

Open teskje opened 2 months ago

teskje commented 2 months ago

The compute introspection sources (in particular mz_dataflow_operator_reachability and the various ones describing dataflow structure) provide almost enough information to compute per-operator frontiers. The only thing missing are intra-operator summaries, which can be approximated by assuming full connectivity for all non-feedback operators, or added by additional logging in Timely. We should consider replacing the current approach of rendering dedicated operators to learn about operator hydration with a WMR view that utilizes our existing logging.

Slack discussion.

teskje commented 1 month ago

One reason why we shouldn't do this is that it makes operator hydration introspection depend on per-replica introspection, with all the usability and availability issues that entails. The operator hydration logging was introduced to fuel hydration progress tracking in the console, and one design decision was specifically to not require replica-targeted queries to retrieve that information.