linkedin / linkedin-gradle-plugin-for-apache-hadoop

Apache License 2.0
117 stars 76 forks source link

Upgrade li-hadoop-plugin for Flow 2.0 #210

Closed reallocf closed 6 years ago

reallocf commented 6 years ago

Introduces LiYamlCompiler. The sole difference between this compiler and the YamlCompiler is how it handles the LiPigBangBang job. This mirrors how the AzkabanDslCompiler and the LiAzkabanDslCompiler are related.

Moves common functions accessed in the AzkabanDslCompiler + YamlCompiler and their Li equivalents into AzkabanCompilerUtils and LiAzkabanCompilerUtils classes.

I also added a unit test to make sure the LiYamlCompiler handles LiPigBangBang jobs correctly.

Other than that this PR adds a little bit of wiring in LiAzkabanPlugin and LiHadoopDslPlugin similar to AzkabanPlugin and HadoopDslPlugin to allow users to toggle between .job/.properties and .flow/.project creation.