search

damballa / parkour

Hadoop MapReduce in idiomatic Clojure.
Apache License 2.0
257 stars 19 forks source link

REPL-launched remote jobs | java.lang.ClassNotFoundException: org.apache.hadoop.util.ShutdownHookManager #2

Closed stanfea closed 10 years ago

stanfea commented 10 years ago

Hi I can't get the example in the doc to run (https://github.com/damballa/parkour/blob/master/doc/repl.md#repl-launched-remote-jobs)

I try to run it like so:

lein repl
(require 'parkour.repl)
(parkour.repl/launch! {"mapred.reduce.tasks" 313} word-count "outpath/0"
         (text/dseq "test.txt"))

Here is the output:

Compiling parkour.core
Exception in thread "main" java.lang.ExceptionInInitializerError, compiling:(core.clj:1:1)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3463)
    at clojure.lang.Compiler.compile1(Compiler.java:7153)
    at clojure.lang.Compiler.compile1(Compiler.java:7143)
    at clojure.lang.Compiler.compile(Compiler.java:7219)
    at clojure.lang.RT.compile(RT.java:398)
    at clojure.lang.RT.load(RT.java:438)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.core$load$fn__5018.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5336)
    at clojure.core$compile$fn__5023.invoke(core.clj:5541)
    at clojure.core$compile.invoke(core.clj:5540)
    at user$eval7.invoke(NO_SOURCE_FILE:1)
    at clojure.lang.Compiler.eval(Compiler.java:6619)
    at clojure.lang.Compiler.eval(Compiler.java:6609)
    at clojure.lang.Compiler.eval(Compiler.java:6582)
    at clojure.core$eval.invoke(core.clj:2852)
    at clojure.main$eval_opt.invoke(main.clj:308)
    at clojure.main$initialize.invoke(main.clj:327)
    at clojure.main$null_opt.invoke(main.clj:362)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:419)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)
Caused by: java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at clojure.lang.RT.loadClassForName(RT.java:2098)
    at clojure.lang.RT.load(RT.java:430)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.core$load$fn__5018.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5336)
    at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
    at clojure.core$load_lib.doInvoke(core.clj:5374)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:619)
    at clojure.core$load_libs.doInvoke(core.clj:5413)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:619)
    at clojure.core$require.doInvoke(core.clj:5496)
    at clojure.lang.RestFn.invoke(RestFn.java:457)
    at parkour.conf$loading__4910__auto__.invoke(conf.clj:1)
    at parkour.conf__init.load(Unknown Source)
    at parkour.conf__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at clojure.lang.RT.loadClassForName(RT.java:2098)
    at clojure.lang.RT.load(RT.java:430)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.core$load$fn__5018.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5336)
    at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
    at clojure.core$load_lib.doInvoke(core.clj:5374)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:619)
    at clojure.core$load_libs.doInvoke(core.clj:5417)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:619)
    at clojure.core$require.doInvoke(core.clj:5496)
    at clojure.lang.RestFn.invoke(RestFn.java:457)
    at parkour.core$loading__4910__auto__.invoke(core.clj:1)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3458)
    ... 26 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ShutdownHookManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)
    at parkour.util.shutdown__init.__init0(Unknown Source)
    at parkour.util.shutdown__init.<clinit>(Unknown Source)
    ... 69 more
Compilation failed: Subprocess failed
SocketException The transport's socket appears to have lost its connection to the nREPL server
Exception in thread "Thread-4" clojure.lang.ExceptionInfo: Subprocess failed {:exit-code 1}
    at clojure.core$ex_info.invoke(core.clj:4327)
    at leiningen.core.eval$fn__3532.invoke(eval.clj:226)
    at clojure.lang.MultiFn.invoke(MultiFn.java:231)
    at leiningen.core.eval$eval_in_project.invoke(eval.clj:326)
    at clojure.lang.AFn.applyToHelper(AFn.java:167)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:619)
    at leiningen.repl$server$fn__7443.invoke(repl.clj:201)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invoke(core.clj:621)
    at clojure.core$bound_fn_STAR_$fn__4102.doInvoke(core.clj:1810)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.lang.Thread.run(Thread.java:744)
    clojure.tools.nrepl.transport/bencode/fn--4287/fn--4288 (transport.clj:95)
    clojure.tools.nrepl.transport/bencode/fn--4287 (transport.clj:95)
    clojure.tools.nrepl.transport/fn-transport/fn--4261 (transport.clj:42)
    clojure.core/binding-conveyor-fn/fn--4107 (core.clj:1836)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:744)
Bye for now!

Here's the error:

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ShutdownHookManager

My project.clj file:

(defproject parkour "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[com.damballa/parkour "0.5.4"]]
  :main parkour.core
  :aot [parkour.core]
  :target-path "target/%s"
  :profiles {
             :dev {:dependencies[[org.clojure/clojure "1.5.1"]]}
             :provided [:hadoop-1-2-1]
             :hadoop-1-2-1 {:dependencies[[org.apache.hadoop/hadoop-core "1.2.1"]]}})

My profiles.clj file:

{:user {:plugins [[lein-hadoop-cluster "0.1.2"]]
        :dependencies [[alembic "0.2.1"]]}}

Here's my leiningen version:

lein --version
Leiningen 2.3.4 on Java 1.7.0_51 OpenJDK 64-Bit Server VM

Running on:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 13.10
Release:    13.10
Codename:   saucy

I've noticed that org.apache.hadoop.util.ShutdownHookManager can be found in hadoop-common but I'm on hadoop 1.2.1 and couldn't find a compatible package.

Thanks!

llasram commented 10 years ago

This looks like a cross-version AOT issue. Parkour dynamically selects which shutdown-hook mechanism to use based on the presence of the class you're getting a not-found exception for. Especially given the presence of an explicit parkour.util.shutdown__init.<clinit> method in your trace, it looks like you have stale AOTed classes from a Hadoop 2-derived distribution.

Doing a lein clean and restarting your REPL should fix things up. You may also need to delete a literal "%s" directory -- I just realized (and acted upon the realization) that I forgot to fix this in my blog post, but the version of Leiningen currently bundled into Alembic doesn't understand profile separation etc, so you need spell out full explicit paths for any non-default :target-path, :compile-path, etc.

stanfea commented 10 years ago

Thanks! that solved it I had tried with hadoop 2.3.0 before going with 1.2.1. lein clean fixed it