search

linkedin / linkedin-gradle-plugin-for-apache-hadoop

Apache License 2.0
117 stars 76 forks source link

fix variable substitution issue when generating YAML files. #228

Closed jamiesjc closed 6 years ago

jamiesjc commented 6 years ago

When adding generateYamlOutput true in the workflow gradle file, the YAML file could not be generated correctly for the hadoopShell job which contains variable substitution "${}"in the usesCommands. When uploading the project zip, it failed because !! was a special character for tags in snakeYaml.

 config:
      command: !!org.codehaus.groovy.runtime.GStringImpl
        metaClass: &id001 !!groovy.lang.MetaClassImpl {}
      command.1: !!org.codehaus.groovy.runtime.GStringImpl
        metaClass: *id001

Did some research and found out the root cause: If the job config contains ${}, which is a GString, it can involve lazy evaluation. It means it's not until the toString() method is invoked that the GString is evaluated. Without toString(), it can cause the incompatible issue when writing the GString directly to YAML file. In CommandJob or HadoopShellJob which extends CommandJob, when calling com.linkedin.gradle.hadoopdsl.job.CommandJob#buildProperties , it will not invoke toString() method for command properties. So it failed when writing to YAML file. There could be other job types that are having the same issue. And the proposed fix applied to all job types.

Reference: https://stackoverflow.com/questions/18975465/groovy-gstringimpl-and-string-behaviour