Retrying Flow stages on failure

[isaacseymour]

First, thanks for this awesome library - it makes parallel FP a joy 😄 . This may just be me being an Elixir newbie, but I'm struggling to work out how to make Flow retry failures, e.g.:

alias Experimental.Flow

flakey_double = fn x ->
  if :rand.uniform(1) == 2, do: raise "oops"
  x * 2
end

flow = 1..1000
       |> Flow.from_enumerable
       |> Flow.map(flakey_double)

Enum.to_list(flow) # => get a few dying GenServers, and then the IEx shell restarts

The same happens if I run the Flow using Flow.start_link(flow), or DynamicSupervisor.start_link([worker(Flow, [])], strategy: :one_for_one) and then DynamicSupervisor.start_child(supervisor_pid, [flow]).

The actual use-case involves hitting the network on each step of a multi-stage flow, so some failures are to be expected, and just restarting the process which is consuming that event should resolve it. Is there any way to get that behaviour using Flow, or should I be looking at using GenStage and a supervision tree?

[josevalim]

@isaacseymour Flow will always fail so you need to control the failures yourself. Either by moving the work elsewhere or by storing failures elsewhere in a way they can be restarted. You probably want to do the latter, using the supervisors for retrying network services is less than ideal because you don't have cool down or back off periods.

[isaacseymour]

Thanks for the speedy response! :)