hadoop 2.6 compile failed

[EricCat]

Same codes works well on hadoop2.4, but when update to hadoop2.6 got the following Error Message:

15/12/30 11:58:56 ERROR parkour.tool: Uncaught exception: clojure.lang.ExceptionInfo: Job tl-crawler.2015-12-22[1/1] failed. {:jname "tl-crawler.2015-12-22[1/1]"}
clojure.lang.ExceptionInfo: Job tl-crawler.2015-12-22[1/1] failed. {:jname "tl-crawler.2015-12-22[1/1]"}
    at clojure.core$ex_info.invoke(core.clj:4327)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at parkour.graph$eval12110$fn__12111$fn__12112.doInvoke(graph.clj:368)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invoke(core.clj:617)
    at parkour.graph$graph_delay$fn__11958.invoke(graph.clj:33)
    at clojure.lang.Delay.deref(Delay.java:33)
    at clojure.core$deref.invoke(core.clj:2128)
    at clojure.core$map$fn__6268.invoke(core.clj:2485)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$apply.invoke(core.clj:617)
    at parkour.graph$graph_delay$fn__11958.invoke(graph.clj:33)
    at clojure.lang.Delay.deref(Delay.java:33)
    at clojure.core$deref.invoke(core.clj:2128)
    at clojure.core$map$fn__6268.invoke(core.clj:2485)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$apply.invoke(core.clj:617)
    at parkour.graph$graph_delay$fn__11958.invoke(graph.clj:33)
    at clojure.lang.Delay.deref(Delay.java:33)
    at clojure.core$deref.invoke(core.clj:2128)
    at parkour.graph$run_graph.invoke(graph.clj:63)
    at parkour.graph$execute.invoke(graph.clj:426)
    at tl_emr.daily.shopstyle.term_total_category$task.invoke(term_total_category.clj:81)
    at tl_emr.daily.shopstyle.term_total_category$tool.doInvoke(term_total_category.clj:88)
    at clojure.lang.RestFn.applyTo(RestFn.java:139)
    at clojure.core$apply.invoke(core.clj:619)
    at parkour.tool.ParkourTool$fn__2383.invoke(tool.clj:33)
    at parkour.tool$integral_STAR_.invoke(tool.clj:12)
    at parkour.tool.ParkourTool.run(tool.clj:31)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at parkour.tool$run.invoke(tool.clj:48)
    at tl_emr.daily.shopstyle.term_total_category$_main.doInvoke(term_total_category.clj:91)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.lang.Var.invoke(Var.java:415)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.main$main_opt.invoke(main.clj:335)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(RestFn.java:436)
    at clojure.lang.Var.invoke(Var.java:423)
    at clojure.lang.AFn.applyToHelper(AFn.java:167)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

term_total_category.clj:81,88,91 looks well, not relate code for error.

My project.clj

:dependencies [[org.clojure/clojure "1.5.1"]
                 [com.damballa/parkour "0.5.4"]
                 [org.apache.avro/avro "1.7.5"]
                 [org.apache.avro/avro-mapred "1.7.5"
                  :classifier "hadoop2"]
                 [transduce/transduce "0.1.1"]
                 [clj-aws-s3 "0.3.10"]
                 [clj-json "0.5.3"]
                 [hadoop-lzo "0.4.15"]
                 ]

:profiles {:provided
             {:dependencies [[org.apache.hadoop/hadoop-client "2.2.0"]
                             [org.apache.hadoop/hadoop-common "2.2.0"]
                             [org.slf4j/slf4j-api "1.6.1"]
                             [org.slf4j/slf4j-log4j12 "1.6.1"]
                             [log4j "1.2.17"]]}
             :test {:resource-paths ["test-resources"]}
             :aot {:aot :all, :compile-path "target/aot/classes"}
             :uberjar {:aot :all}
             :jobjar [:aot]})

The error messages is very wried. I can't find out where should be fixed.

leiningen version is 2.5.3 jdk is 1.7.0_91

[llasram]

That's the exception which just means that a job failed; at least when I last checked there wasn't a way to get anything more detailed back client-side than "the job failed for some reason." You'll need to look at the cluster-side logs to see why the individual tasks failed.

FWIW my cluster is running also running Hadoop 2.6 with OpenJDK 7, so I don't believe there to be a fundamental incompatibility anywhere.

[EricCat]

@llasram Thanks for the quick reply