TIL_10_July_2017 - Fault Tolerance and Supervisor in Elixir

[vinhnglx]

• The aim of fault tolerance is to acknowledge the existence of failures, minimize their impact and ultimately recover without human intervention.

Regardless of which part of the system happens to fail, you shouldn't take down the entire system. Your priority should be to minimize their effects, isolate errors and recover from them automatically.

Fault Tolerance is a first-class concept in Erlang/Elixir. Below are basic techniques of detecting and handling errors in a concurrent system.

[vinhnglx]

There are three types of runtime errors: errors, exits and throw.

• errors

  • Few typical examples of errors
  • Raise an error by raise
  • Append ! to each function that you want to explicitly raises an error. This is a convention in Elixir.

iex(1)> 1/0
** (ArithmeticError) bad argument in arithmetic expression
    :erlang./(1, 0)
iex(1)> Hello.nonexistence_function
** (UndefinedFunctionError) function Hello.nonexistence_function/0 is undefined (module Hello is not available)
    Hello.nonexistence_function()
iex(1)> {1,2,3,4} |> List.first
** (FunctionClauseError) no function clause matching in List.first/1
    (elixir) lib/list.ex:219: List.first({1, 2, 3, 4})
iex(1)> raise "hell"
** (RuntimeError) hell

• exit is used to deliberately terminate a process.

# without exit
iex(2)> spawn(fn ->
...(2)> IO.puts "Whew"
...(2)> end)
Whew
#PID<0.93.0>

# with exit
iex(1)> spawn(fn ->
...(1)> exit("I'm done")
...(1)> IO.puts "Whew"
...(1)> end)
#PID<0.89.0>

• throw allows non-local returns. When you are deep in a loop, throw a value and catch it up the call stack is the way to stop the loop and return a value - shouldn't use it

[vinhnglx]

• Use try construct to intercept and kind of run-time error and do something about it.

try do
   ....
catch error_type, error_value ->
  ...
end

• error_type will contain an atom :error, :exit, or :throw.
• catch is a pattern match, multiple clauses can be specified.
• after block is useful to clean up resources

[vinhnglx]

• In general, a runtime error has a type, which can be: error, exit or throw. A runtime error has a value. If the runtime error isn't handled, the corresponding process will terminate.

[vinhnglx]

How to handle the error in the concurrent system.

Because processes share no memory, a crash in one process won't leave memory garbage that might corrupt another process.

But, in fact, processes often communicate with each other. Therefore, we have a way of detecting a process crash and somehow recovering from it.

• Linking process

If two process are linked and one of them terminates, the other process receives an exit signal - a notification that a process has crashed.

An exit signal contains the PID of the crashed process and the exit reason. By default, when a process receives an exit signal from another process, the linked process terminates as well.

One link connects exactly two processes and is always bidirectional.

Use Process.link/1: Creates a link between the calling process and another process. Or we can use spawn_link which spawns a process and links it to the current one.

One process can be linked to an arbitrary number of other processes. Hence, if the default behavior isn't overridden, then those processes will crash as well

[vinhnglx]

• To detect process crash and do something about it, we can use trapping exits.

When a process is trapping exits, it isn’t taken down when a linked process crashes. Instead, an exit signal is placed in the surviving process's message queue, in the form of a standard message. A trapping process can receive these messages and do something about the crash.

Call Process.flag/2 or Process.flag/3 to setup an exit trap.

[vinhnglx]

• Links are always bidirectional. In some cases, unidirectional propagation of a process crash works better.

Use Process.monitor/1: the calling process starts monitoring the given item. It returns the monitor reference.

A single process can create multiple monitors.

[vinhnglx]

There are two main differences between monitors and links.

• First, monitors are unidirectional - only the process that created a monitor receives notifications
• Unlike a link, the observer process won’t crash when the monitored process terminates. Instead, a message is sent, which you can handle or ignore.

[vinhnglx]

Supervisor

The idea behind Supervisor is simple. Each worker process is supervised by a supervisor process. Whenever a worker process terminates, the supervisor starts another one in its place. The supervisor basically does nothing but supervise, which makes its code simple, error-free and thus unlikely to crash.

A Supevisor work-follow:

• Run the supervisor process
• Supervisor process traps exits.
• From the within the supervisor process, child processes are started and linked to the supervisor process.
• If a crash happens, the supervisor process receives an exit signal and performs corrective actions: such as restart the crashed process.
• If a supervisor is terminated, child processes are terminated immediately.

[vinhnglx]

• Supervisor knows which processes must be started by callback module - the one you write and plug into the behaviour.