search

iconara / rubydoop

Write Hadoop jobs in JRuby
220 stars 33 forks source link

Could not load job setup script ("word-count") #35

Open janpieper opened 9 years ago

janpieper commented 9 years ago

If I try to run the word-count example, I keep getting the following error and I don't know how to get it running...

$ git clone git@github.com:iconara/rubydoop.git
$ cd rubydoop/examples/word_count
$ bundle install
$ rake package
$ hadoop jar build/word_count.jar word-count conf/hadoop-local.xml README.md output                                                                        
LoadError: no such file to load -- word-count
  require at org/jruby/RubyKernel.java:1065
  require at file:/tmp/hadoop-unjar2289886275803509919/lib/jruby-complete-1.7.12.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
Exception in thread "main" rubydoop.RubydoopRunnerException: Could not load job setup script ("word-count"): "(LoadError) no such file to load -- word-count"
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:36)
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:18)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at rubydoop.RubydoopJobRunner.main(RubydoopJobRunner.java:50)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.jruby.embed.InvokeFailedException: (LoadError) no such file to load -- word-count
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:249)
        at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1369)
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:33)
        ... 10 more
Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file to load -- word-count
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1065)
        at RUBY.require(file:/tmp/hadoop-unjar2289886275803509919/lib/jruby-complete-1.7.12.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55)
$ ruby -v
jruby 1.7.12 (1.9.3p392) 2014-04-15 643e292 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 +indy [linux-amd64]
jowl commented 9 years ago

It seems as if you're not running the latest Rubydoop, but follow the latest instructions? Either pull origin/master and re-run your commands or try running

$ hadoop jar build/word_count.jar word_count_job path/to/input path/to/output

which I think is how the previous version worked.

Having that said, it appears as if the instructions in examples/word_count/README.md are a bit off; you shouldn't create the output directory and ../README.md should probably be README.md, unless I'm missing something. But I'm guessing that you've already figured that out as your commands look sensible, apart from the part that runs Hadoop.

janpieper commented 9 years ago

I've changed my Gemfile to use the latest code:

gem "rubydoop",
  :git => "git@github.com:iconara/rubydoop.git",
  :branch => "master"

Now I get the following errors:

NoMethodError: undefined method `run' for Rubydoop:Module
  (root) at /tmp/hadoop-unjar1573602031273691108/META-INF/app.home/bin/word-count:10
    load at org/jruby/RubyKernel.java:1081
  (root) at classpath:jar-bootstrap.rb:14
    each at org/jruby/RubyArray.java:1613
  (root) at classpath:jar-bootstrap.rb:10
jowl commented 9 years ago

Ah, now I think I understand. You must have checked out the latest version of Rubydoop, but not re-built lib/rubydoop.jar. You should revert your word count Gemfile to use the path dependency like it did before, git won't work because the aforementioned JAR file isn't checked in, and then run rake build in the project's root folder (i.e. rubydoop/ in your setup). This will re-build the JAR file which should enable you to run the examples using the same commands as you described first.

janpieper commented 9 years ago 
$ cd rubydoop
$ git checkout v2.0.0.pre0
$ bundle install
$ rake build
mkdir -p ext/build
rake aborted!
No such file or directory - /home/jan/workspace/rubydoop/.classpath
org/jruby/RubyIO.java:3663:in `foreach'
/home/jan/workspace/rubydoop/Rakefile:14:in `(root)'
org/jruby/RubyBasicObject.java:1533:in `instance_eval'
/home/jan/workspace/rubydoop/Rakefile:12:in `(root)'
org/jruby/RubyProc.java:271:in `call'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:246:in `execute'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:184:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101:in `top_level'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101:in `top_level'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:73:in `run'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
org/jruby/RubyKernel.java:1081:in `load'
org/jruby/RubyKernel.java:1101:in `eval'
/home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15:in `(root)'
Tasks: TOP => build => build:jars => build:compile => build:setup
(See full trace by running task with --trace)
janpieper commented 9 years ago

Was able to generate a .classpath file, but the build still breaks...

$ rake setup:hadoop
$ rake setup:classpath
$ rake build
$ rake build
mkdir -p ext/build
rake aborted!
Unable to find a javac compiler;
com.sun.tools.javac.Main is not on the classpath.
Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/java-8-oracle/jre"
org.apache.tools.ant.taskdefs.compilers.CompilerAdapterFactory.getCompiler(org/apache/tools/ant/taskdefs/compilers/CompilerAdapterFactory.java:129)
org.apache.tools.ant.taskdefs.Javac.findSupportedFileExtensions(org/apache/tools/ant/taskdefs/Javac.java:979)
org.apache.tools.ant.taskdefs.Javac.scanDir(org/apache/tools/ant/taskdefs/Javac.java:956)
org.apache.tools.ant.taskdefs.Javac.execute(org/apache/tools/ant/taskdefs/Javac.java:927)
org.apache.tools.ant.UnknownElement.execute(org/apache/tools/ant/UnknownElement.java:292)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:497)
java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
RUBY.(root)(/home/jan/workspace/rubydoop/Rakefile:21)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:271)
RUBY.execute(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:246)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.execute(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:241)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:184)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:170)
RUBY.invoke_task(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:143)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101)
RUBY.run_with_threads(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:110)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:95)
RUBY.run(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:73)
RUBY.standard_exception_handling(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:160)
RUBY.run(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:70)
org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1081)
org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:1101)
java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
home.jan.$_dot_rvm.gems.jruby_minus_1_dot_7_dot_12.bin.jruby_executable_hooks.(root)(/home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15)
home.jan.$_dot_rvm.gems.jruby_minus_1_dot_7_dot_12.bin.jruby_executable_hooks.(root)(home/jan/$_dot_rvm/gems/jruby_minus_1_dot_7_dot_12/bin//home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15)
Tasks: TOP => build => build:jars => build:compile
(See full trace by running task with --trace)
iconara commented 9 years ago

Perhaps JAVA_HOME does not point to the JDK. It is currently set to "/usr/lib/jvm/java-8-oracle/jre"

You don't have a JDK installed, only a JRE. You can find JDK 8 here: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

However, there's no need to do this if you just want to write your own Rubydoop jobs. It looks like you just want to use the latest version, and if that is the case there's a prerelease of it on Rubygems: gem 'rubydoop', '= 2.0.0.pre0'

I'm not sure you want to use the latest version, though. Unless you absolutely need packaging with Puck you should use the latest stable version, 1.2.0.

janpieper commented 9 years ago

I think I got it working... These are the steps that was needed to get the word_count example running.

$ git clone git@github.com:iconara/rubydoop.git
$ cd rubydoop
$ git checkout v2.0.0.pre0
$ bundle install
$ rake setup:hadoop
$ rake setup:classpath
$ JAVA_HOME=/usr/lib/jvm/jdk1.8.0_60 rake build
$ cd examples/word_count
$ bundle install
$ rake package
$ hadoop jar build/word_count.jar word-count -conf conf/hadoop-local.xml README.md output

Now there is a output directory which contains the following files:

part-r-00000
.part-r-00000.crc
_SUCCESS
._SUCCESS.crc
hudsonsferreira commented 9 years ago

@janpieper I followed the same steps of you and it didn't worked.

➜  word_count git:(55aa17c) ✗ hadoop jar build/word_count.jar word-count -conf conf/hadoop-local.xml README.md output
LoadError: no such file to load -- rubydoop
                              require at org/jruby/RubyKernel.java:939
                              require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
                                <top> at uri:classloader:/META-INF/gem.home/rubydoop-2.0.0.pre0/lib/rubydoop.rb:4
                              require at org/jruby/RubyKernel.java:939
                               (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
                                <top> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
                                 load at org/jruby/RubyKernel.java:957
  block in classpath:jar-bootstrap.rb at classpath:jar-bootstrap.rb:14
                                 each at org/jruby/RubyArray.java:1560
                                <top> at classpath:jar-bootstrap.rb:10
iconara commented 9 years ago

Both of you: why are you building your own version, why are you not installing the gem?

hudsonsferreira commented 9 years ago

@iconara Because this is how the example dir is. https://github.com/iconara/rubydoop/blob/master/examples/word_count/Gemfile