linkedin / linkedin-gradle-plugin-for-apache-hadoop

Apache License 2.0
117 stars 76 forks source link

Add conditional workflow feature. #222

Closed jamiesjc closed 6 years ago

jamiesjc commented 6 years ago

This is part of the conditional workflow design: https://github.com/azkaban/azkaban/issues/1826

We need to generate the condition section for each job or subflow that has user-defined conditions. Example: Define the condition for JobB in the workflow block.

workflows.gradle

workflow('AZTest') {
  commandJob('JobA') {
    uses 'bash ./sample_script.sh'
  }

  commandJob('JobB') {
    uses 'echo "If param1 in JobA is 1, run JobB."'
    depends 'JobA'
    conditions '${JobA:param1} == 1'
   }

  targets 'JobB'
}
reallocf commented 6 years ago

Maybe we should add a check to see if a root level flow sets a condition. So have it throw a WARN if a root level workflow condition is set so users know that it won't take effect.

Adding the warning would be in this file: https://github.com/linkedin/linkedin-gradle-plugin-for-apache-hadoop/blob/master/hadoop-plugin/src/main/groovy/com/linkedin/gradle/hadoopdsl/checker/WorkflowJobChecker.groovy Then you can add a simple negative test case that creates a workflow and tries to set a root level condition to confirm that the output contains the proper warning.

Let me know if you have any questions about this! I think it will make it more straightforward for users 👍

reallocf commented 6 years ago

Talked offline about adding additional conditions. @jamiesjc said she'd do this in the next commit.

Also need to update VERSIONS.md and CONTRIBUTORS.md.

jamiesjc commented 6 years ago

Thanks for the review @reallocf Todo in the next PR:

  1. Add WARN checker for workflow root level condition.
  2. Add ERROR checker for invalid conditions.