OpenFn / kit

The bits & pieces that make OpenFn work. (diagrammer, cli, compiler, runtime, runtime manager, logger, etc.)
8 stars 12 forks source link

Engine: support multiple inputs to a step #704

Closed josephjclark closed 4 weeks ago

josephjclark commented 1 month ago

This PR enables the runtime call each step multiple times.

Each time a step is called, it passes its output to any downstream steps, and so the downstream steps are run multiple times.

It basically treats a repeated step as if it were a unique step in the workflow. So if step x is run twice, the runtime basically runs x-0 and x-1 as if there were defined on the plan. Note that actually call x-0 x, because otherwise 8 million tests break. And I think it's a bit cleaner?

Issue

Closes #698

Changes

Important Behavioural notes

These are the rules of this feature:

josephjclark commented 1 month ago

Hey @mtuchi is there any chance you can run a test of this for me?

You'll need to install openfnx from this branch and run a workflow through the CLI. It should work for any workflow where you want to call the same step from multiple places

mtuchi commented 1 month ago

@josephjclark my initial test passed 👍🏽, i want to test important behaviors that you described in depth i will share more feedback later today