search

mesos / hadoop

Hadoop on Mesos
176 stars 80 forks source link

Getting hadoop-mesos to work on older hadoop #51

Closed hansbogert closed 9 years ago

hansbogert commented 9 years ago

I'm trying to run this project on older mapreduce, 1.0.x. If it were up to me I wouldn't be using this but I have to. I've already successfully set this up under the recommended 2.5.0 version and I had no problems at all, so this is probably something specific to 1.0.x.

So, trying to get this to work eventually leads me to the following error/log on a mesos-executor

cat /local/vdbogert/var/lib/mesos/slaves/20150420-104917-33592586-5050-17145-S0/frameworks/20150424-154625-234919178-5050-2360-0000/executors/executor_Task_Tracker_97/runs/latest/stderr
I0424 15:47:49.596091  9570 exec.cpp:132] Version: 0.21.0
I0424 15:47:49.610085  9586 exec.cpp:206] Executor registered on slave 20150420-104917-33592586-5050-17145-S0
15/04/24 15:47:49 INFO mapred.MesosExecutor: Executor registered with the slave
15/04/24 15:47:49 INFO mapred.MesosExecutor: Launching task : Task_Tracker_97
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobConf.writeXml(Ljava/io/Writer;)V
        at org.apache.hadoop.mapred.MesosExecutor.configure(MesosExecutor.java:48)
        at org.apache.hadoop.mapred.MesosExecutor.launchTask(MesosExecutor.java:80)
Exception in thread "Thread-1" I0424 15:47:49.731550  9586 exec.cpp:413] Deactivating the executor libprocess

Has anybody got this to work under older hadoop distro's? If not, can someone estimate how much work it would be, solving problems like the above.

And as last question, how would I solve the above error.

Thanks

tarnfeld commented 9 years ago

Getting this to work with the old versions of hadoop shouldn't be that much of a pain. We're making use of most classes in the mapred namespace rather than mapreduce which means that changes in 2.x over 1.x are pretty minor. That said, the JobConf/Configuration classes and some others do come from the 2.x code so there are going to be some method incompatibilities.

As far as I know it's not going to be possible to have the same version of the code run against both 1.x and 2.x because classes have been moved and renamed all over the place. I would suggest iterating over the error messages you see and finding workarounds and then moving forward to see how much work it'll be.

Just to make sure, even though this is based around 2.5.0 (CDH5 to be specific) the API is still purely MapReduce v1 and should be completely compatible with any 1.x jobs you have. Though I appreciate I don't know what the reasons for your 1.x limitation are :smile:

hansbogert commented 9 years ago

Commenting out the offending part in source gets everything working on hadoop 1.0.0. I'll try to get a fix merged when I have the time, the error originated in debug code, so no prio for now.