search

ScriptFUSION / Porter

:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
GNU Lesser General Public License v3.0
611 stars 24 forks source link

Durability is broken for subsequent generator iterations after the first #31

Closed Bilge closed 6 years ago

Bilge commented 7 years ago

Durability is provided for the $provider->fetch call, but Provider::fetch is declared to return Iterator, which is typically implemented using generators. Generators imply deferred code executions, which means that even if the generator throws an exception, it is not caught by the retry handler because it already exited that code block.

This common case is not captured by PorterTest because it only tests that Provider::fetch throws an exception directly instead of the generator throwing an exception.

Bilge commented 7 years ago

Workaround available since 17345ba. However, this is a crude stop-gap solution because it only supports the first iteration of the generator. Resources may reuse a connector on subsequent iterations: a typical case is seamless pagination where the subsequent page requests may fail but are still not covered by Porter's durability features.

To fully resolve this issue we need to provide durability features at the connector level as in v1, but unlike those versions we will not require connectors to duplicate durability code in each implementation. Porter will provide a factory which accepts a closure from the connector when it makes connections and provides preconfigured durability features for that connection. This feature will target Porter v4.

Bilge commented 6 years ago

Solved with ConnectionContext::retry introduced in #46 :label: 4.0.0. Connector implementations are expected to wrap their fetch code in a closure passed to this retry() method.