linkedin / linkedin-gradle-plugin-for-apache-hadoop

Apache License 2.0
117 stars 76 forks source link

Fix base properties scope for subflows when generating YAML files. #226

Closed jamiesjc closed 6 years ago

jamiesjc commented 6 years ago

The issue happens when defining baseProperties for jobs inside subflows when generating YAML files. For example, in workflow.gradle:

hadoop {
  buildPath "jobs/embeddedFlowWithBaseProperties"
  cleanPath false
  generateYamlOutput true

  workflow('embeddedFlow') {
    propertySet('common') {
      set properties: ['user.to.proxy' : 'foo']
    }

    workflow('embeddedFlow1') {
      commandJob('shellEcho') {
        baseProperties 'common'
        uses 'echo "This is an echoed text from embeddedFlow1."'
      }
      targets 'shellEcho'
    }

    targets 'embeddedFlow1'
  }
}

This would throw below exception:

Caused by: java.lang.NullPointerException: Cannot invoke method fillProperties() on null object
        at com.linkedin.gradle.hadoopdsl.job.Job.buildProperties(Job.groovy:161)
        at com.linkedin.gradle.hadoopdsl.job.CommandJob.super$2$buildProperties(CommandJob.groovy)
        at com.linkedin.gradle.hadoopdsl.job.CommandJob.buildProperties(CommandJob.groovy:59)
        at com.linkedin.gradle.hadoopdsl.job.CommandJob$buildProperties.call(Unknown Source)
        at com.linkedin.gradle.hadoopdsl.job.Job$buildProperties$0.call(Unknown Source)
        at com.linkedin.gradle.azkaban.AzkabanDslYamlCompiler.yamlizeJob(AzkabanDslYamlCompiler.groovy:305)

The issue was caused by the incorrect parent scope of subflows when recursively yamlizing the subflow. The scope needs to be updated during the recursion.