This commit contains two changes that probably can be split up into two separate PRs that can be applied to the base branch. I'm putting this up as a draft for now so @PaulHax can get a sense of the changes.
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using evaluateChain and asyncSelect.
evaluateChain is responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source.
To keep processing a data source like how the old pipeline code supported nested executions, evaluateChain is invoked inside a loop for every data source. asyncSelect is used to drive the loop execution, seleting evaluateChain promises whenever they are done.
The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.
This commit contains two changes that probably can be split up into two separate PRs that can be applied to the base branch. I'm putting this up as a draft for now so @PaulHax can get a sense of the changes.
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using
evaluateChainandasyncSelect.evaluateChainis responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source.To keep processing a data source like how the old pipeline code supported nested executions,
evaluateChainis invoked inside a loop for every data source.asyncSelectis used to drive the loop execution, seletingevaluateChainpromises whenever they are done.The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.