tarantool / queue

Create task queues, add and take jobs, monitor failed tasks
Other
236 stars 52 forks source link

Add master-replica switching support #165

Closed 0x501D closed 2 years ago

0x501D commented 2 years ago

This patch adds ability to use queue in a master-replica scheme. The queue will monitor the operation mode of the tarantool and perform the necessary actions accordingly.

This patch adds five states for queue: INIT, STARTUP, RUNNING, ENDING and WAITING. When the tarantool is launched for the first time, the state of the queue is always INIT until box.info.ro is false.

States switching scheme:

stateDiagram-v2
direction LR
w: Waiting
note right of w
  monitor rw status
end note
s: Startup
note left of s
  wait for maximum
  upstream lag * 2
  release all tasks
  start driver
end note
Init --> s
r: Running
note right of r
  queue is ready
end note
e: Ending
note left of e
  stop driver
end note
w --> s: ro -> rw
s --> r
r --> e: rw -> ro
e --> w

In the STARTUP state, the queue is waiting for possible data synchronization with other cluster members by the time of the largest upstream lag multiplied by two. After that, all taken tasks are released, except for tasks with session uuid matching inactive sessions uuids. This makes possible to take a task, switch roles on the cluster, and release the task within the timeout specified by the queue.cfg({ttr = N}) parameter. Note: all clients that take() and do not ack()/release() tasks must be disconnected before changing the role. And the last step in the STARTUP state is starting tube driver using new method called start(). Each tube driver must implement start() and stop() methods. The start() method should start the driver fibers, if any, in other words, initialize all asynchronous work with the tubes space. The stop() methods shuld stop the driver fibers, if any, respectively.

In the RUNNING state, the queue is working as usually. The ENDING state calls stop() method. in the WAITING state, the queue listens for a change in the read_only flag.

All states except INIT is controlled by new fiber called 'queue_state_fiber'.

A new release_all() method has also been added, which forcibly returns all taken tasks to a ready state. This method can be called per tube.

Closes #120

LeonidVas commented 2 years ago

Mark all commits after Closes as Follows up.