Open chyezh opened 1 month ago
what about name it as streaming service?
Version 2.4 relies on the pub/sub capability of MQ for both reading and writing paths to support data persistence and querying of streaming data respectively.
Write path
Read path
TimeTick lifetime
In version 2.5, the pub/sub API is provided by StreamNode, and MQ reading and writing are encapsulated within StreamNode. Write path
Read path
TimeTick lifetime Significant changes in dependency order between versions 2.4 and 2.5 imply that the existing upgrade plan cannot meet the requirements.
Upgrade MixCoord, including RootCoord, QueryCoord, DataCoord, StreamCoord, at this time:
Stop dataNode, the flush process will be terminated, Proxy can still accept all requests.
Upgrade Proxy, once all proxies are upgraded:
Pros and cons:
Version 2.5 servers as a transitional version to 3.0, now we can take plan2 to ensure a smooth upgrade from 2.4, making code cleanup after upgrading to 3.0.
Make sure upgrade can be smoothly is very important.
To simply the work we need to do, maybe we can keep delegator at querynode and do not move it to stream service.
One problem is how many stream node need to upgrade and it's size.
To upgrade smoothly, streaming node need to assign timestamp and try to merge data from TTstream and Proxy insert.
in 2.5 we can keep delegator still at querynode. and move delegator to streaming node at 3.0
Motivation
Architecture
The following changes will be made:
Components Responsibility
Goals
RoadMap
Streaming Service Implementation
Use Streaming Service To Produce
Use Streaming Service To Consume In Query
Use Streaming Service To Consume In Flush
Rolling upgrade
Incoming
Limitation