The issue happens when defining baseProperties for jobs inside subflows when generating YAML files. For example, in workflow.gradle:
hadoop {
buildPath "jobs/embeddedFlowWithBaseProperties"
cleanPath false
generateYamlOutput true
workflow('embeddedFlow') {
propertySet('common') {
set properties: ['user.to.proxy' : 'foo']
}
workflow('embeddedFlow1') {
commandJob('shellEcho') {
baseProperties 'common'
uses 'echo "This is an echoed text from embeddedFlow1."'
}
targets 'shellEcho'
}
targets 'embeddedFlow1'
}
}
This would throw below exception:
Caused by: java.lang.NullPointerException: Cannot invoke method fillProperties() on null object
at com.linkedin.gradle.hadoopdsl.job.Job.buildProperties(Job.groovy:161)
at com.linkedin.gradle.hadoopdsl.job.CommandJob.super$2$buildProperties(CommandJob.groovy)
at com.linkedin.gradle.hadoopdsl.job.CommandJob.buildProperties(CommandJob.groovy:59)
at com.linkedin.gradle.hadoopdsl.job.CommandJob$buildProperties.call(Unknown Source)
at com.linkedin.gradle.hadoopdsl.job.Job$buildProperties$0.call(Unknown Source)
at com.linkedin.gradle.azkaban.AzkabanDslYamlCompiler.yamlizeJob(AzkabanDslYamlCompiler.groovy:305)
The issue was caused by the incorrect parent scope of subflows when recursively yamlizing the subflow. The scope needs to be updated during the recursion.
The issue happens when defining baseProperties for jobs inside subflows when generating YAML files. For example, in workflow.gradle:
This would throw below exception:
The issue was caused by the incorrect parent scope of subflows when recursively yamlizing the subflow. The scope needs to be updated during the recursion.