search

mesos / hadoop

Hadoop on Mesos
176 stars 80 forks source link

Can not launch TaskTracker (Error occurred during initialization of VM) #67

Open RecursionTaoist opened 9 years ago

RecursionTaoist commented 9 years ago

I run wordcount demo use hadoop-2.5.0-cdh5.3.2 (hadoop-mapreduce1-project) + mesos-0.23.0-rc4 Found “launched but no heartbeat yet” in jobTracker log all the time in task's stdout:

CPLUS_INCLUDE_PATH=/opt/lib/boost_1_58_0
MANPATH=/opt/lib/mvapich2.2/share/man:/opt/compiler/gcc-4.8.2/man:/usr/share/man
HOSTNAME=dn-137-211
...
HISTSIZE=1000
HADOOP_HOME=/home/hadoop/hadoop2
HADOOP_DEV_HOME=/home/hadoop/hadoop2
LIBRARY_PATH=/opt/compiler/gcc-4.8.2/lib64:/opt/lib/cuda-6.5/lib
MESOS_DIRECTORY=/home/mesos/slave/slaves/20150918-160900-3549014208-5050-14584-S0/frameworks/20150923-170913-3549014208-5050-6111-0008/executors/executor_Task_Tracker_0/runs/39ece48d-1f91-446b-ae49-72a0a6f66346
FPATH=/opt/lib/mvapich2.2/include
OLDPWD=/home/mesos/slave/slaves/20150918-160900-3549014208-5050-14584-S0/frameworks/20150923-170913-3549014208-5050-6111-0008/executors/executor_Task_Tracker_0/runs/39ece48d-1f91-446b-ae49-72a0a6f66346
SSH_TTY=/dev/pts/3
LC_ALL=C
USER=root
.....
LD_LIBRARY_PATH=/opt/lib/mvapich2.2/lib:/opt/lib/mvapich2.2/lib/shared:/opt/lib/liblmdb-0.9/lib:/opt/lib/protobuf-2.5/lib:/opt/lib/gflag-1.4.0/lib:/opt/lib/glog-0.3.3/lib:/opt/lib/boost_1_58_0/stage/lib:/opt/lib/opencv-2.4.9/lib:/opt/lib/log4cplus-1.2.0-rc3/lib:/opt/tool/intel/lib/intel64:/opt/tool/intel/mkl/lib/intel64:/opt/compiler/gcc-4.8.2/lib64:/opt/lib/cuda-6.5/lib64
MESOS_EXECUTOR_ID=executor_Task_Tracker_0
CPATH=/opt/lib/mvapich2.2/include:/opt/lib/liblmdb-0.9/include:/opt/lib/protobuf-2.5/include:/opt/lib/gflag-1.4.0/include:/opt/lib/glog-0.3.3/include:/opt/lib/opencv-2.4.9/include:/opt/lib/log4cplus-1.2.0-rc3/include:/opt/lib/cuda-6.5/include
HADOOP_MAPARED_HOME=/home/hadoop/hadoop2
PATH=/home/hadoop/spark/bin:/home/hadoop/spark/sbin:/home/hadoop/spark/lib:/opt/scheduler/mesos-0.23.0-rc4/libexec/mesos:/opt/scheduler/mesos-0.23.0-rc4/libexec:/opt/scheduler/mesos-0.23.0-rc4/bin:/opt/scheduler/mesos-0.23.0-rc4/sbin:/opt/scheduler/mesos-0.23.0-rc4/lib:/home/hadoop/hadoop2/sbin:/home/hadoop/hadoop2/bin:/opt/tool/git-2.4.5/bin:/opt/tool/git-2.4.5/bin:....6.5/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/hadoop/pig/bin:/opt/scheduler/mesos/bin:/opt/scheduler/mesos/sbin:/root/bin:/home/hadoop/pig/bin:/opt/scheduler/mesos/bin:/opt/scheduler/mesos/sbin:/home/hadoop/pig/bin:/opt/scheduler/mesos/bin:/opt/scheduler/mesos/sbin:/home/hadoop/pig/bin:/opt/scheduler/mesos/bin:/opt/scheduler/mesos/sbin:/home/hadoop/pig/bin:/opt/scheduler/mesos/bin:/opt/scheduler/mesos/sbin
HDFS_CONF_DIR=/home/hadoop/hadoop2/etc/hadoop
MESOS_HOME=/home/mesos
HADOOP_HDFS_HOME=/home/hadoop/hadoop2
PWD=/home/mesos/slave/slaves/20150918-160900-3549014208-5050-14584-S0/frameworks/20150923-170913-3549014208-5050-6111-0008/executors/executor_Task_Tracker_0/runs/39ece48d-1f91-446b-ae49-72a0a6f66346/hadoop-2.5.0-cdh5.3.2
HADOOP_COMMON_HOME=/home/hadoop/hadoop2
F90=gfortran
MESOS_NATIVE_JAVA_LIBRARY=/opt/scheduler/mesos/lib/libmesos-0.23.0.so
JAVA_HOME=/opt/lib/jdk
MESOS_NATIVE_LIBRARY=/opt/scheduler/mesos/lib/libmesos.so
HADOOP_CONF_DIR=/home/hadoop/hadoop2/etc/hadoop
HADOOP_OPTS=-Xmx4096m -XX:NewSize=1365m -XX:MaxNewSize=2457m
MESOS_SLAVE_PID=slave(1)@192.168.137.211:5051
MESOS_FRAMEWORK_ID=20150923-170913-3549014208-5050-6111-0008
MESOS_PATH=/opt/scheduler/mesos-0.23.0-rc4
MESOS_CHECKPOINT=0
SHLVL=2
HOME=/root
LIBPROCESS_PORT=0
YARN_CONF_DIR=/home/hadoop/hadoop2/etc/hadoop
MESOS_SLAVE_ID=20150918-160900-3549014208-5050-14584-S0
MODULESHOME=/usr/share/Modules
HADOOP_BIN=/home/hadoop/hadoop2/bin
...
_=/bin/env
Error occurred during initialization of VM
Too small initial heap for new size specified```
RecursionTaoist commented 9 years ago

I modify the code, use config "mapred.mesos.executor.jvm.opts" take place of defaultOpts + taskTrackerJVMHeap as TaskTracker's $HADOOP_OPTS, then TT can launch successfully, but I think this is Not a good solution

tarnfeld commented 9 years ago

It'd be useful to know what memory configuration you'd set for the slots, and how many slots this task tracker had. I imagine the sum total in this case was too low and there's some tuning to be done?