search

risingwavelabs / risingwave

SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.
https://www.risingwave.com/slack
Apache License 2.0
6.57k stars 536 forks source link

RFC: Unify schema and pk derivation in backend #2063

Open TennyZhuang opened 2 years ago

TennyZhuang commented 2 years ago

Currently, we do PK derivation in frontend, but how to access schema and pk_indices in backend? There are two ways in our repo:

  1. Pass the schema (and pk_indices) to backend node in StreamNode::fields.
  2. Derive the schema in backend use the same logic, such as generate_agg_schema.

Currently, most of executors use the second way to calculate the schema, but use the first way to achieve pk_indices.

Pros of the first way:

We don't need to do the same thing twice, and it's safe that the results are always consistent.

Cons of the first way:

The derivation of most operators are simple enough, pass them to backend is wasteful. e.g. Limit, Filter, Hop.

Another solution is:

  1. Derivation in backend, but share the same codes. In detail, we can associate backend ExecutorBuilder and frontend LogicalNode, create a temporal graph consists of LogicalNode, and call the same derivation codes in backend.

Cons of third way: How to resolve dependency between crates?

Which one is better?

st1page commented 2 years ago

+1 for share the same codes. and we can extrate the XXXCore in https://github.com/singularity-data/risingwave/issues/1373 to crate::common::sql

liurenjie1024 commented 2 years ago

Prefer 1. For example when we rolling upgrade a large cluster, the logic may change in different version.

TennyZhuang commented 2 years ago

Prefer 1. For example when we rolling upgrade a large cluster, the logic may change in different version.

It’s impossible to modify schema and pk_indices derivation rule without compatibility. In fact, it’s just a formula based on relational algebra.

Will you concern that 1+1=2 may change during rolling upgrade?

Another problem is that we always have multiple frontends and multiple compute nodes, so it’s still necessary to keep the compatibility.

TennyZhuang commented 2 years ago

And even for other logicals that may change, the correct solution is that only schedule new request to compute nodes with matched version, so we can simplify many things.

TennyZhuang commented 2 years ago

And for high-concurrent severing query, the schema is much bigger than the response body.

TennyZhuang commented 2 years ago

So I’d prefer 3

liurenjie1024 commented 2 years ago

Prefer 1. For example when we rolling upgrade a large cluster, the logic may change in different version.

It’s impossible to modify schema and pk_indices derivation rule without compatibility. In fact, it’s just a formula based on relational algebra.

Will you concern that 1+1=2 may change during rolling upgrade?

Another problem is that we always have multiple frontends and multiple compute nodes, so it’s still necessary to keep the compatibility.

Yes, we should not allow this case.