search

azkaban / azkaban-plugins

Plugins for Azkaban.
https://azkaban.github.io
Apache License 2.0
130 stars 178 forks source link

class org/apache/hadoop/conf/Configuration not found #242

Open shadyxu opened 7 years ago

shadyxu commented 7 years ago

I installed the CDH Hadoop client on the executor by yum, compiled the Azkaban-plugin project(version 3.0.0) by ant, and configured

hadoop.home=/usr/lib/hadoop
jobtype.global.classpath=${hadoop.home}/*,/etc/hadoop/conf

in commonprivate.properties. /usr/lib/hadoop is where the Hadoop client is installed and /etc/hadoop/conf is where the Hadoop configuration xml files are.

However, when I try to start the executor, Azkaban still reports this error: Exception in thread "main" azkaban.jobtype.JobTypeManagerException: azkaban.jobtype.JobTypeManagerException: azkaban.jobExecutor.utils.JobExecutionException: java.lang.IllegalStateException: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at azkaban.jobtype.JobTypeManager.loadPlugins(JobTypeManager.java:82) at azkaban.jobtype.JobTypeManager.<init>(JobTypeManager.java:65) at azkaban.execapp.FlowRunnerManager.<init>(FlowRunnerManager.java:202) at azkaban.execapp.AzkabanExecutorServer.<init>(AzkabanExecutorServer.java:143) at azkaban.execapp.AzkabanExecutorServer.main(AzkabanExecutorServer.java:331) Caused by: azkaban.jobtype.JobTypeManagerException: azkaban.jobExecutor.utils.JobExecutionException: java.lang.IllegalStateException: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at azkaban.jobtype.JobTypeManager.loadPluginJobTypes(JobTypeManager.java:168) at azkaban.jobtype.JobTypeManager.loadPlugins(JobTypeManager.java:79) ... 4 more Caused by: azkaban.jobExecutor.utils.JobExecutionException: java.lang.IllegalStateException: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at azkaban.jobtype.JobTypeManager.loadJobTypes(JobTypeManager.java:238) at azkaban.jobtype.JobTypeManager.loadPluginJobTypes(JobTypeManager.java:164) ... 5 more Caused by: java.lang.IllegalStateException: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at azkaban.utils.Utils.getCause(Utils.java:252) at azkaban.utils.Utils.callConstructor(Utils.java:285) at azkaban.utils.Utils.callConstructor(Utils.java:269)

I checked and found the exact org.apache.hadoop.conf.Configuration class in /usr/lib/hadoop/hadoop-common.jar. Am I doing anything wrong or is there a bug?

shadyxu commented 7 years ago

I did some tests and found that classpath do not accept * as placeholder. Adding all the jars one by one is annoying, trying and finding which jars are needed sucks. Any good idea?

suvodeep-pyne commented 7 years ago

Classpath does accept /path/to/foo/* for jars. This should work. We use it for our systems.

dirkdaems commented 7 years ago

I can confirm I encounter the same issue. Also when looking at the code, I see nothing that would resolve the asterisk.

wangqiaoshi commented 7 years ago

i encounter the same issue too.

wangqiaoshi commented 7 years ago

@suvodeep-pyne how do you configure? i use * don't work,but some.jar work