When a source crashes, e.g. the process exits, dataloader currently stores the source under the name :error. Reading a key from a failed source thus raises an error such as:
** (RuntimeError) Source does not exist: :assoc
Registered sources are:
[:error]
Approach
With this PR,
a failed source is kept, as there might still be loaded data in it
a separate field failed_sources in Dataloader stores which sources failed, and the reason
Dataloader.get/4 on a failed source first tries to fetch the key from the source. If that fails, the error for the entire source is returned instead of the individual key error
failed sources are retried on next Dataloader.run/1
Assumptions
All sources are implemented in a way so that Source.fetch on a missing key returns an error tuple. Otherwise, Dataloader.get/4 could first check whether a source failed and not call Source.fetch in this case, removing the ability to get previously (successfully) loaded data.
Source failure does not need to be handled outside Dataloader. Otherwise, we could raise an exception, or make Dataloader.run/1 return ok/error tuples, maybe when an option is set.
Failed sources are retryable.
Please let me know what you think. If and when we merge this to master, I'd also be keen to open a PR targeting the v1 branch 👍
When a source crashes, e.g. the process exits, dataloader currently stores the source under the name
:error
. Reading a key from a failed source thus raises an error such as:Approach
With this PR,
failed_sources
inDataloader
stores which sources failed, and the reasonDataloader.get/4
on a failed source first tries to fetch the key from the source. If that fails, the error for the entire source is returned instead of the individual key errorDataloader.run/1
Assumptions
Source.fetch
on a missing key returns anerror
tuple. Otherwise,Dataloader.get/4
could first check whether a source failed and not callSource.fetch
in this case, removing the ability to get previously (successfully) loaded data.Dataloader
. Otherwise, we could raise an exception, or makeDataloader.run/1
returnok
/error
tuples, maybe when an option is set.Please let me know what you think. If and when we merge this to
master
, I'd also be keen to open a PR targeting thev1
branch 👍