MaterializeInc / materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.
https://materialize.com
Other
5.72k stars 466 forks source link

storage,compute: unify timely clusters #17416

Open benesch opened 1 year ago

benesch commented 1 year ago

Part of the multipurpose clusters epic (#17413).

As part of the initial cluster unification work (https://github.com/MaterializeInc/cloud/issues/4929), we unified storaged and computed into a single clusterd process, but left the storage and compute timely clusters separate. That is, a clusterd process hosts two timely clusters: one for storage and one for compute.

To support multipurpose clusters, we need to unify these two timely clusters. There are many design questions to shake out:

cc @guswynn @aljoscha @antiguru @teskje

guswynn commented 1 year ago

More details in a braindump here: https://www.notion.so/materialize/ComputeState-StorageState-Unification-f8139794f9e449408ecd5244f4ae814b