search

azkaban / azkaban-plugins

Plugins for Azkaban.
https://azkaban.github.io
Apache License 2.0
130 stars 178 forks source link

Issue while Running Pig Job - Latest Azakaban #269

Open boycode opened 7 years ago

boycode commented 7 years ago

Running on HDP.

I was able to create the command,java jobtypes. but while running the pig job I am facing an hadoopsecurity . Struggling with this issue.

21-06-2017 21:32:33 IST pig INFO - Command: /home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/execute-as-user platadmin java -Dazkaban.flowid=pig -Dazkaban.execid=84 -Dazkaban.jobid=pig -Xms64M -Xmx256M -cp /home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/lib/azkaban-common-0.1.0-SNAPSHOT.jar:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/pig/azkaban-jobtype-3.0.0.jar:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/lib/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.jar:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig/_resources_pig:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/pig/pig-0.12.0-withouthadoop.jar:/usr/hdp/2.4.2.0-258/pig/lib/:/usr/hdp/2.4.2.0-258/pig/:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/pig/$HADOOP_HOME/conf:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/pig/$HADOOP_HOME/lib/:/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/pig/lib/ azkaban.jobtype.HadoopSecurePigWrapper -logfile /home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig/piglogfile454710543297702496.log /home/platadmin/azkabanpath/upload.pig 21-06-2017 21:32:33 IST pig INFO - Environment variables: {JOB_OUTPUT_PROP_FILE=/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig/pig_output_3809097616448187317_tmp, JOB_PROP_FILE=/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig/pig_props_4953922702774292643_tmp, KRB5CCNAME=/tmp/krb5ccA6_Sample_Pigpigpig84__platadmin, PIG_LOG_FILE=/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig/piglogfile454710543297702496.log, JOB_NAME=pig} 21-06-2017 21:32:33 IST pig INFO - Working directory: /home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/executions/84/A6_Sample_Pig 21-06-2017 21:32:33 IST pig INFO - Changing user: user: platadmin, uid: 1018, gid: 1018 21-06-2017 21:32:33 IST pig INFO - user command starting from: java 21-06-2017 21:32:33 IST pig INFO - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Logger 21-06-2017 21:32:33 IST pig INFO - at azkaban.jobtype.HadoopSecurePigWrapper.(HadoopSecurePigWrapper.java:53)

My Settings are the following

1) Copied the pig-0.12.0 and created a symlink pig. 2) plugin.properties pig.listener.visualizer=false jobtype.classpath=${pig.home}/lib/,${pig.home}/ pig.home=$PIG_HOME 3)private.properties jobtype.classpath=${hadoop.home}/conf,${hadoop.home}/lib/,lib/ jobtype.class=azkaban.jobtype.HadoopPigJob 4) Common.properties hadoop.home=$HADOOP_HOME pig.home=$PIG_HOME execute.as.user=true 5) hadoop.home=$HADOOP_HOME pig.home=$PIG_HOME execute.as.user=true azkaban.native.lib=/home/platadmin/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes

pig.job

type=pig pig.script=/home/platadmin/azkabanpath/upload.pig pig.home=/usr/hdp/2.4.2.0-258/pig obtain.binary.token=false user.to.proxy=platadmin

This is the only settings done. Can you please help

Thanks, Vishnu

boycode commented 7 years ago

https://stackoverflow.com/questions/44805783/issue-while-running-pig-job-latest-azkaban

Keris commented 6 years ago

Hi @boycode , could you tell me the version of plugin you used? I have problems with building the latest plugin. Thanks in advance.