[WIP] Prototype workflow builder interface

[cfournie]

Here's what I'd like to use as an API for the workflow builder that allows people to specify:

• Actions that they want executed (and their dependencies or other metadata), and
• Transformations of the workflow execution graph using those actions and the action metadata specified.

This would be the final PR merged to add the XML workflow generation feature, before this there will need to be PRs that:

1. Add a basic Workflow tag/graph to pyoozie.tags; and
2. Add some basic transformation functions (and the graph functions required to implement them).

For an initial review, I'd like design comments and questions.

[honkfestival]

I think this would be simpler if you remove the concept of a layout entirely and only support actions with dependencies.

Does it really matter how the actions are laid out if all the dependencies are met? A topological sort on actions with their dependencies as edges should suffice.

[cfournie]

I think this would be simpler if you remove the concept of a layout entirely and only support actions with dependencies.

At least one action must have no dependencies (as an initial action). I'm trying to offer a way for someone to specify:

• work actions (i.e. actions that produce something that you care about between which there are dependencies that manifest as edges specified by a layout); and
• decorator actions (i.e. actions, edges, and other control-flow nodes such as fork/join/kill that determine how to act in error situations including the insertion of notification actions).

Does it really matter how the actions are laid out if all the dependencies are met? A topological sort on actions with their dependencies as edges should suffice.

There are two main ways that we can lay out work actions:

• Sequentially using a topological sort as you suggested; or
• As a fork-join DAG that introduces parallel execution at times.

The fork-join DAG could be lain out in a wide variety of ways; we'll provide a naive implementation that relies upon just dependency information to create the graph but you may want to define your own strategy which may:

• Take into account median job runtime to rearrange the workflow to minimize the critical path;
• Multiply schedule the same job within a workflow and use external synchronization to minimize the critical path.

Both are strategies that we've discussed investigating.

[honkfestival]

Sequentially using a topological sort as you suggested; or

Using topological sort doesn't require that we run the jobs sequentially. Rosetta Code has an example in Python of how to sort so that you know which groups can be run in parallel.

• Take into account median job runtime to rearrange the workflow to minimize the critical path;

Minimizing the critical path necessarily means breaking up the DAG, which would mean breaking one flow into multiple flows. I don't see how manually specifying fork/join points would help (but I'm probably missing some context).

• Multiply schedule the same job within a workflow and use external synchronization to minimize the critical path.

Sorry, but I don't understand this. Let's chat IRL to make sure I'm not missing something.

[cfournie]

Talked IRL and I'll begin prototyping a different approach.