Open hzxa21 opened 3 months ago
- Find the first aligned barrier in source and transform it into Pause barrier to trigger scaling immediately.
- Make scaling to not relying on barrier (@wenym1 can comment more one the details).
I believe https://github.com/risingwavelabs/risingwave/issues/13396 can be addressed by adopting a very similar idea if we find it feasible. BTW, it is possible now to share more on the details?
When barrier latency is high due to insufficient parallelism, user may want to scale their streaming jobs accordingly when resources are sufficient to accelerate computation.
The current implementation of scaling has the following properties:
This results in a dilemma:
This makes me think that it is a flaw in the current scaling mechanism and we should improve it. Some ideas after discussion with @wenym1: