search

DigitalPebble / behemoth

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
Other
281 stars 60 forks source link

ClassNotFoundException org.apache.mahout.math.Vector #29

Closed telekoma closed 12 years ago

telekoma commented 12 years ago

Hello,

I tried out behemoth and walked through the tutorial successfully until I wanted to create the vectors for Mahout.

When running the SparseVectorsFromBehemoth command the hadoop Job fails after several ClassNotFoundExceptions

These classes are "org.apache.mahout.math.Vector" and "org.apache.mahout.math.function.ObjectLongProcedure"

I am using hadoop 1.0.0.

Hope this isn't the wrong place for my request. Didn't find any information for an mailing list or some other place for support :-)

jnioche commented 12 years ago

Hi, we've changed the dependency management from Ivy to Maven so maybe things are behaving a bit differently. I'll have a look. The mailing list for Behemoth related questions is : http://groups.google.com/group/digitalpebble?pli=1

telekoma commented 12 years ago

Thanks for the response. I worked around the problem by saving the content of the lib directory in the mahout job jar directly to the lib dirs of the tasktrackers.

jnioche commented 12 years ago

Am able to reproduce the issue. We have to force a dependency on Mahout collections which for some reason is not pulled. The main problem however is that we call the Mahout jobs from within the jars and they in turn are not able to locate the dependencies properly e.g. they do not call job.setJarByClass(this.getClass()); I will look at ways of avoiding this, in the worst case we'd have to duplicate the Mahout jobs and put them in our packages

jnioche commented 12 years ago

Fixed in [master 8cff4ee] The Mahout libs are now unpacked in the job file + declaring single dependency on mahout-core. I also updated the code so that it works with Mahout 0.5 Please give it a try and close the issue if it is fixed. Note : I was getting an empty mahout-math jar and had to delete it from my local cache

telekoma commented 12 years ago

Yay! Works perfectly now. Big thanks!