runabol / piper

piper - a distributed workflow engine
Apache License 2.0
489 stars 86 forks source link

adding control flow types #41

Closed anshumanr closed 5 years ago

anshumanr commented 5 years ago

Hi, Piper control flow seems to be missing flow control types such as loop & next. Actually a loop can be executed by a switch & next kind of control types. Any suggestions on how to add a next kind of control or any other means to achieve that? Thanks.

runabol commented 5 years ago

Did you check out each and map?

anshumanr commented 5 years ago

Yes, each doesn't sound like a loop as it's done in parallel. map is as maps are. next type of functionality is what i need. Unless i am mistaken, none of the existing constructs seem to be able to do that.

runabol commented 5 years ago

Maybe you can describe your use case.

anshumanr commented 5 years ago

Use case is a usual programmatic flow.

Task 1 -> Task 2 (if condition then Task 5 else Task 3) -> Task 3 -> Task 4 -> End
            \-> Task 5 -> Task 6 -> Task 7 (if condition then Task 3 else Task 5)

Basically depending on conditions I may want to either jump ahead some tasks or go back to a previous task in the flow. switch allows me to do conditional branching but ability to alter flow by jumping to a task is missing.

From what i have understood so far, the DefaultTaskCompletionHandler handle() gets the next task to execute. A simple approach would be to use a next field to jump to a particular task based on task id.

Any thoughts on this approach?

anshumanr commented 5 years ago

That's the change basically, and seems to work. Of course, next is working of internal task ids (starting with 0) and any change there would break this.

@@ -61,6 +61,7 @@ public class DefaultTaskCompletionHandler implements TaskCompletionHandler {
   @Override
   public void handle (TaskExecution aTask) {
     log.debug("Completing task {}, {}", aTask.getId(), jobRepository.toString());
+    int next = aTask.getInteger("next", -1);
     Job job = jobRepository.findJobByTaskId (aTask.getId());
     if(job!=null) {
       SimpleTaskExecution task = SimpleTaskExecution.createForUpdate(aTask);

@@ -74,7 +75,10 @@ public class DefaultTaskCompletionHandler implements TaskCompletionHandler {
         contextRepository.push(job.getId(), newContext);
       }
       if(hasMoreTasks(mjob)) {
-        mjob.setCurrentTask(mjob.getCurrentTask()+1);
+        if (next > 0)
+          mjob.setCurrentTask(next);
+        else
+          mjob.setCurrentTask(mjob.getCurrentTask()+1);
         jobRepository.merge(mjob);
         jobExecutor.execute(mjob);
       }

The sample script


label: My Pipeline

inputs:
  - name: name
    type: string
    required: true

tasks:      
  - label: Print a greeting
    type: print
    text: Hello ${name}
    next: 2

  - name: randomNumber
    label: Generate a random number
    type: randomInt
    startInclusive: 0
    endInclusive: 5000
    next: 4

  - name: randomNumber2
    label: Generate a random number
    type: randomInt
    startInclusive: 5000
    endInclusive: 15000

  - label: Print a farewell
    type: print
    text: Is this task skipped ${name}
    next: 1

  - label: Print a farewell
    type: print
    text: Goodbye ${name}
runabol commented 5 years ago

That looks a lot like a goto. Not something I'd want to introduce. Seems like an if/else support would be more suitable for what you're trying to do.

anshumanr commented 5 years ago

switch gives us if/else but doesn't give me the control I want. For now I will stick with my solution.

PS: just fyi, another flow control language is the Amazon state language, which does exactly the same with next