Lifecycle States & Transitions

thedrow commented 3 years ago

State Machine

I defined an hierarchical state machine to model the lifecycle of the actor. Other states and transitions between them may be added in the future.

The transitions library allows you to append callbacks to the different events of the state machine. Please refer to their documentation if you'd like to find out which callbacks are available and their order of execution.

The following diagram defines the fundamental actor states and how to transition between them: Actor Lifecycle State Machine

States

The state machine has states which define where the actor is during his lifecycle.

The following tables describes all current states:

Actor State Machine

State Name	Description
`initializing`	The initial state of the actor when created.
`initialized`	The state of the actor when we start it for the very first time
`starting`	The state of the actor immediately after calling `actor.start()`
`starting -> dependencies_started`	The state of the actor after all of the actor's dependencies have been started
`starting -> resources_acquired`	The state of the actor after all resources have been acquired
`starting -> tasks_started`	The state of the actor after all tasks have been started
`started`	The state of the actor after it has finished all startup activities
`started -> running`	The state of the actor when all tasks are running
`started -> paused`	The state of the actor when all tasks are halted without shutting down the entire actor
`started -> running -> healthy`	The state of the actor when the actor is functioning properly
`started -> running -> degraded`	The state of the actor when the actor is not functioning properly but is still able to perform some of its duties
`started -> running -> unhealthy`	The state of the actor when the actor is temporarily not functioning
`stopping`	The state of the actor immediately after calling `actor.stop()`
`stopping -> tasks_stopped`	The state of the actor after all tasks have been stopped
`stopping -> resources_released`	The state of the actor after all resources have been released
`stopping -> dependencies_stopped`	all of the actor's dependencies have been stopped
`stopped`	The state of the actor after it has finished all shutdown activities
`crashed`	The state of the actor after an exception was raised during startup or shutdown

Actor Restart State Machine

State Name	Description
`ignore`	A special state which is ignored by the Actor
`restarting`	The state of the actor once it has begun restarting
`restarting -> stopping`	The state of the actor while stopping during a restart
`restarting -> starting`	The state of the actor while starting during a restart
`restarted`	The state of the actor after it has been restarted

TODOs

[x] The crashed state is currently not in use, it is pending the completion of https://github.com/pytransitions/transitions/issues/485 or another solution.
[x] There should be a way to restart the actor, modeled as a state.

Triggers

Triggers are the API the state machine exposes to trigger a transition between states.

The following triggers are already defined:	Trigger Name	Description
`initialize()`	Initializes the actor without starting it
`start()`	Starts the actor
`pause()`	Pauses the actor's tasks without shutting it down completely
`resume()`	Resumes the actor's tasks after it has been paused
`stop()`	Stops the actor
`restart()`	Restarts the actor
`report_error()`	Report that an error has occurred while starting or stopping the actor
`report_warning()`	Report an issue with the actor which interferes with some of the actor's functionality
`report_problem()`	Report an issue with the actor which causes the actor to be temporarily malfunctioning
`recover()`	Recover from a degraded or unhealthy states

Transitions

The after transition callback allows us to progress the state machine until the desired state by repeatedly calling the start() or stop() triggers until the desired state is reached. We use the before transition callbacks to manage the lifecycle of the actor.

The following table describe what the current side effects from transitions:	From State	To State
`starting`	`starting -> dependencies_started`	Before this transition, all the actor's dependencies are transitioned to the `started` state
`starting -> dependencies_started`	`starting -> resources_acquired`	Before this transition, all the actor's resources are acquired
`starting -> resources_acquired`	`starting -> tasks_started`	Before this transition, all the actor's tasks are started
`stopping -> tasked_stopped`	`stopping -> resources_released`	Before this transition, all the actor's resources are released
`stopping -> resources_released`	`stopping -> dependencies_stopped`	Before this transition, all the actor's dependencies are transitioned to the `stopped` state

The rest of the transitions have no use currently other than stepping into the next step. In the future we will introduce a public API to let the users use them.

TODOs

[x] The transition between stopped and starting requires test coverage - #11.
[x] Sometimes actor can recover from crashes. We should define a transition between crashed and starting using the start() trigger.

MicahLyle commented 3 years ago

@thedrow

The crashed state is currently not in use, it is pending the completion of pytransitions/transitions#485 or another solution.

At the time of this comment, you have that unchecked, but it looks like the issue is resolved. Is crashed in use or intended to be in use?

thedrow commented 3 years ago

We need to use the new on_exception hook instead of rolling our own implementation.

celery / jumpstarter