search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Hadoop 1.2.1, HiBench 3.0.0 and Mahout 0.7 compatible? #85

Open echozyw opened 9 years ago

echozyw commented 9 years ago

Hi there,

I would like to check if HiBench 3.0.0 is compatible with Hadoop 1.2.1? I notice the document of HiBench mentioned that HiBench is tested against Hadoop 1.0.4 and 2.2.0. What about Hadoop 1.2.1? 

I have issues when running HiBench 3.0.0 againt Hadoop 1.2.1. Am wondering if this might be the issue?

Thanks a lot in advance! Gina

adrian-wang commented 9 years ago

HiBench 3.0 should support hadoop 1.2.1, thought we may not tested that.

echozyw commented 9 years ago

Thank you for the quick response. I appreciate it! Here are some warnings which occur very often when I ran ./run.sh for the Bayes workload. Could you shed light on what might go wrong here:

15/04/03 02:58:41 INFO mapred.JobClient: map 0% reduce 0% 15/04/03 03:12:09 INFO mapred.JobClient: Task Id : attempt_201504022127_0012_m_000003_0, Status : FAILED Error launching task 15/04/03 03:12:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0012_m_000003_0&filter=stdout 15/04/03 03:12:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0012_m_000003_0&filter=stderr 15/04/03 03:12:09 INFO mapred.JobClient: Task Id : attempt_201504022127_0012_m_000004_0, Status : FAILED Error launching task 15/04/03 03:12:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0012_m_000004_0&filter=stdout 15/04/03 03:12:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0012_m_000004_0&filter=stderr 15/04/03 03:12:57 INFO mapred.JobClient: map 6% reduce 0% ....

adrian-wang commented 9 years ago

Can you share your hibench-config?

echozyw commented 9 years ago

No problem.

this="${BASH_SOURCE-$0}" bin=$(cd -P -- "$(dirname -- "$this")" && pwd -P) script="$(basename -- "$this")" this="$bin/$script"

export HIBENCH_VERSION="3.0"

###################### Global Paths ##################

export JAVA_HOME=

export HADOOP_HOME=

export HADOOP_EXECUTABLE=

export HADOOP_CONF_DIR=

export HADOOP_EXAMPLES_JAR=

export MAPRED_EXECUTABLE=

Set the varaible below only in YARN mode

export HADOOP_JOBCLIENT_TESTS_JAR=

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64 export HADOOP_HOME=/home/fedora/hadoop-1.2.1 export HADOOP_EXECUTABLE=/home/fedora/hadoop-1.2.1/bin/hadoop export HADOOP_CONF_DIR=/home/fedora/hadoop-1.2.1/conf export HADOOP_EXAMPLES_JAR=/home/fedora/hadoop-1.2.1/hadoop-examples-1.2.1.jar export MAPRED_EXECUTABLE=

Set the varaible below only in YARN mode

export HADOOP_JOBCLIENT_TESTS_JAR=

export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_VERSION=hadoop2 # set it to hadoop1 to enable MR1, hadoop2 to enable MR2

if $HADOOP_EXECUTABLE version|grep -i -q cdh4; then HADOOP_RELEASE=cdh4 HADOOP_VERSION=cdh4 elif $HADOOP_EXECUTABLE version|grep -i -q cdh5; then HADOOP_RELEASE=cdh5 HADOOP_VERSION=cdh5 elif $HADOOP_EXECUTABLE version|grep -i -q "hadoop 2"; then HADOOP_RELEASE=hadoop2 HADOOP_VERSION=hadoop2 else HADOOP_RELEASE=hadoop1 HADOOP_VERSION=hadoop1 fi

if [ "x"$HADOOP_VERSION == "xhadoop1" ]; then

CONFIG_REDUCER_NUMBER=mapred.reduce.tasks CONFIG_MAP_NUMBER=mapred.map.tasks else

CONFIG_REDUCER_NUMBER=mapreduce.job.reduces CONFIG_MAP_NUMBER=mapreduce.job.maps fi

echo JAVA_HOME=${JAVA_HOME:? "ERROR: Please set paths in $this before using HiBench."} echo HADOOP_HOME=${HADOOP_HOME:? "ERROR: Please set paths in $this before using HiBench."} echo HADOOP_EXECUTABLE=${HADOOP_EXECUTABLE:? "ERROR: Please set paths in $this before using HiBench."} echo HADOOP_CONF_DIR=${HADOOP_CONF_DIR:? "ERROR: Please set paths in $this before using HiBench."} echo HADOOP_EXAMPLES_JAR=${HADOOP_EXAMPLES_JAR:? "ERROR: Please set paths in $this before using HiBench."}

echo MAPRED_EXECUTABLE=${MAPRED_EXECUTABLE:? "ERROR: Please set paths in $this before using HiBench."}

if [ -z "$HIBENCH_HOME" ]; then export HIBENCH_HOME=dirname "$this"/.. fi

if [ -z "$HIBENCH_CONF" ]; then export HIBENCH_CONF=${HIBENCH_HOME}/conf fi

if [ -f "${HIBENCH_CONF}/funcs.sh" ]; then . "${HIBENCH_CONF}/funcs.sh" fi

if [ -z "$DEPENDENCY_DIR" ]; then export DEPENDENCY_DIR=${HIBENCH_HOME}/common/hibench fi

if [ -z "$HIVE_HOME" ]; then export HIVE_RELEASE=hive-0.12.0-bin export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_RELEASE} fi

if [ -z "$MAHOUT_HOME" ]; then export MAHOUT_RELEASE=mahout-distribution-0.7 export MAHOUT_EXAMPLE_JOB="mahout-examples-0.7-job.jar" export MAHOUT_HOME=${DEPENDENCY_DIR}/mahout/target/${MAHOUT_RELEASE} fi

if [ -z "$NUTCH_HOME" ]; then export NUTCH_RELEASE=nutch-1.2 export NUTCH_HOME=${DEPENDENCY_DIR}/nutchindexing/target/${NUTCH_RELEASE} fi

if [ -z "$DATATOOLS" ]; then export DATATOOLS=${HIBENCH_HOME}/common/autogen/dist/datatools.jar fi

if [ $# -gt 1 ] then if [ "--hadoop_config" = "$1" ] then shift confdir=$1 shift HADOOP_CONF_DIR=$confdir fi fi

base dir HDFS

export DATA_HDFS=/HiBench

export DATA_HDFS=/home/fedora

local report

export HIBENCH_REPORT=${HIBENCH_HOME}/hibench.report

################# Compress Options #################

swith on/off compression: 0-off, 1-on.

Switch it off (COMPRESS_GLOBAL=0) for better performance

export COMPRESS_GLOBAL=1 export COMPRESS_CODEC_GLOBAL=org.apache.hadoop.io.compress.DefaultCodec export COMPRESS_CODEC_MAP=org.apache.hadoop.io.compress.DefaultCodec

Set COMPRESS_CODEC_MAP to SnappyCodec (as shown below) for better performance

export COMPRESS_CODEC_MAP=org.apache.hadoop.io.compress.SnappyCodec

export COMPRESS_CODEC_GLOBAL=com.hadoop.compression.lzo.LzoCodec

export COMPRESS_CODEC_GLOBAL=org.apache.hadoop.io.compress.SnappyCodec

echozyw commented 9 years ago

Also, here is how it crashed when I ./run.sh for the Bayes workload. Thanks a lot!

[fedora@newjobtestmaster bin]$ ./run.sh ========== running bayes bench ========== Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64 HADOOP_HOME=/home/fedora/hadoop-1.2.1 HADOOP_EXECUTABLE=/home/fedora/hadoop-1.2.1/bin/hadoop HADOOP_CONF_DIR=/home/fedora/hadoop-1.2.1/conf HADOOP_EXAMPLES_JAR=/home/fedora/hadoop-1.2.1/hadoop-examples-1.2.1.jar

done from hibench-config.sh

Warning: $HADOOP_HOME is deprecated.

Deleted hdfs://192.168.111.240:9000/home/fedora/Bayes/Output-comp Warning: $HADOOP_HOME is deprecated.

MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Warning: $HADOOP_HOME is deprecated.

Running on hadoop, using /home/fedora/hadoop-1.2.1/bin/hadoop and HADOOP_CONF_DIR=/home/fedora/hadoop-1.2.1/conf MAHOUT-JOB: /home/fedora/HiBench/common/hibench/mahout/target/mahout-distribution-0.7/mahout-examples-0.7-job.jar Warning: $HADOOP_HOME is deprecated.

15/04/03 03:17:57 INFO vectorizer.SparseVectorsFromSequenceFiles: Maximum n-gram size is: 3 15/04/03 03:17:57 INFO vectorizer.SparseVectorsFromSequenceFiles: Minimum LLR value: 1.0 15/04/03 03:17:57 INFO vectorizer.SparseVectorsFromSequenceFiles: Number of reduce tasks: 24 15/04/03 03:22:02 INFO input.FileInputFormat: Total input paths to process : 48 15/04/03 03:22:03 INFO mapred.JobClient: Running job: job_201504022127_0013 15/04/03 03:22:04 INFO mapred.JobClient: map 0% reduce 0% 15/04/03 03:22:07 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000049_0, Status : FAILED Error launching task 15/04/03 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000049_0&filter=stdout 15/04/03 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000049_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000001_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000001_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000001_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000003_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000003_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000003_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000004_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000005_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000005_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000005_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000006_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000006_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000006_0&filter=stderr 15/04/03 03:22:18 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000007_0, Status : FAILED Error launching task 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000007_0&filter=stdout 15/04/03 03:22:18 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000007_0&filter=stderr 15/04/03 03:22:30 INFO mapred.JobClient: map 2% reduce 0% 15/04/03 03:22:39 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000001_1, Status : FAILED Error launching task 15/04/03 03:22:39 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000001_1&filter=stdout 15/04/03 03:22:39 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000001_1&filter=stderr 15/04/03 03:22:42 INFO mapred.JobClient: map 3% reduce 0% 15/04/03 03:22:45 INFO mapred.JobClient: map 4% reduce 0% 15/04/03 03:22:50 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000004_1, Status : FAILED Error launching task 15/04/03 03:22:50 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_1&filter=stdout 15/04/03 03:22:50 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_1&filter=stderr 15/04/03 03:23:03 INFO mapred.JobClient: map 5% reduce 0% 15/04/03 03:23:03 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000004_2, Status : FAILED Error launching task 15/04/03 03:23:03 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_2&filter=stdout 15/04/03 03:23:03 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000004_2&filter=stderr 15/04/03 03:23:04 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000003_1, Status : FAILED Error launching task 15/04/03 03:23:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000003_1&filter=stdout 15/04/03 03:23:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000003_1&filter=stderr 15/04/03 03:23:07 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000005_1, Status : FAILED Error launching task 15/04/03 03:23:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000005_1&filter=stdout 15/04/03 03:23:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000005_1&filter=stderr 15/04/03 03:23:14 INFO mapred.JobClient: map 6% reduce 0% 15/04/03 03:23:17 INFO mapred.JobClient: Task Id : attempt_201504022127_0013_m_000006_1, Status : FAILED Error launching task 15/04/03 03:23:17 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000006_1&filter=stdout 15/04/03 03:23:17 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0013_m_000006_1&filter=stderr 15/04/03 03:23:26 INFO mapred.JobClient: map 7% reduce 0% 15/04/03 03:23:30 INFO mapred.JobClient: map 8% reduce 0% 15/04/03 03:23:44 INFO mapred.JobClient: map 9% reduce 0% 15/04/03 03:23:50 INFO mapred.JobClient: map 10% reduce 0% 15/04/03 03:23:56 INFO mapred.JobClient: map 11% reduce 0% 15/04/03 03:24:05 INFO mapred.JobClient: map 12% reduce 0% 15/04/03 03:24:16 INFO mapred.JobClient: map 13% reduce 0% 15/04/03 03:24:28 INFO mapred.JobClient: map 14% reduce 0% 15/04/03 03:24:52 INFO mapred.JobClient: map 15% reduce 0% 15/04/03 03:25:01 INFO mapred.JobClient: map 16% reduce 0% 15/04/03 03:25:04 INFO mapred.JobClient: map 17% reduce 0% 15/04/03 03:25:13 INFO mapred.JobClient: map 18% reduce 0% 15/04/03 03:25:27 INFO mapred.JobClient: map 19% reduce 0% 15/04/03 03:25:34 INFO mapred.JobClient: map 20% reduce 0% 15/04/03 03:25:39 INFO mapred.JobClient: map 21% reduce 0% 15/04/03 03:25:46 INFO mapred.JobClient: map 22% reduce 0% 15/04/03 03:26:07 INFO mapred.JobClient: map 24% reduce 0% 15/04/03 03:26:10 INFO mapred.JobClient: map 25% reduce 0% 15/04/03 03:26:22 INFO mapred.JobClient: map 26% reduce 0% 15/04/03 03:26:25 INFO mapred.JobClient: map 27% reduce 0%

15/04/03 03:26:41 INFO mapred.JobClient: map 28% reduce 0%

15/04/03 03:26:53 INFO mapred.JobClient: map 30% reduce 0% 15/04/03 03:27:08 INFO mapred.JobClient: map 31% reduce 0% 15/04/03 03:27:11 INFO mapred.JobClient: map 32% reduce 0% 15/04/03 03:27:20 INFO mapred.JobClient: map 33% reduce 0% 15/04/03 03:27:31 INFO mapred.JobClient: map 34% reduce 0% 15/04/03 03:27:36 INFO mapred.JobClient: map 35% reduce 0% 15/04/03 03:27:43 INFO mapred.JobClient: map 36% reduce 0% 15/04/03 03:27:48 INFO mapred.JobClient: map 37% reduce 0% 15/04/03 03:28:06 INFO mapred.JobClient: map 38% reduce 0% 15/04/03 03:28:13 INFO mapred.JobClient: map 39% reduce 0% 15/04/03 03:28:18 INFO mapred.JobClient: map 40% reduce 0% 15/04/03 03:28:25 INFO mapred.JobClient: map 41% reduce 0% 15/04/03 03:28:39 INFO mapred.JobClient: map 42% reduce 0% 15/04/03 03:28:45 INFO mapred.JobClient: map 43% reduce 0% 15/04/03 03:28:51 INFO mapred.JobClient: map 44% reduce 0% 15/04/03 03:28:57 INFO mapred.JobClient: map 45% reduce 0% 15/04/03 03:29:11 INFO mapred.JobClient: map 46% reduce 0% 15/04/03 03:29:23 INFO mapred.JobClient: map 47% reduce 0% 15/04/03 03:29:30 INFO mapred.JobClient: map 49% reduce 0% 15/04/03 03:30:00 INFO mapred.JobClient: map 51% reduce 0% 15/04/03 03:30:15 INFO mapred.JobClient: map 53% reduce 0% 15/04/03 03:30:30 INFO mapred.JobClient: map 54% reduce 0% 15/04/03 03:30:33 INFO mapred.JobClient: map 55% reduce 0% 15/04/03 03:30:45 INFO mapred.JobClient: map 56% reduce 0% 15/04/03 03:31:04 INFO mapred.JobClient: map 57% reduce 0% 15/04/03 03:31:06 INFO mapred.JobClient: map 58% reduce 0% 15/04/03 03:31:16 INFO mapred.JobClient: map 59% reduce 0% 15/04/03 03:31:19 INFO mapred.JobClient: map 60% reduce 0% 15/04/03 03:31:33 INFO mapred.JobClient: map 61% reduce 0% 15/04/03 03:31:42 INFO mapred.JobClient: map 62% reduce 0% 15/04/03 03:31:45 INFO mapred.JobClient: map 63% reduce 0% 15/04/03 03:31:54 INFO mapred.JobClient: map 64% reduce 0% 15/04/03 03:32:12 INFO mapred.JobClient: map 65% reduce 0% 15/04/03 03:32:19 INFO mapred.JobClient: map 66% reduce 0% 15/04/03 03:32:24 INFO mapred.JobClient: map 67% reduce 0% 15/04/03 03:32:31 INFO mapred.JobClient: map 68% reduce 0% 15/04/03 03:32:43 INFO mapred.JobClient: map 69% reduce 0% 15/04/03 03:32:49 INFO mapred.JobClient: map 70% reduce 0% 15/04/03 03:32:55 INFO mapred.JobClient: map 71% reduce 0% 15/04/03 03:33:00 INFO mapred.JobClient: map 72% reduce 0% 15/04/03 03:33:17 INFO mapred.JobClient: map 74% reduce 0% 15/04/03 03:33:28 INFO mapred.JobClient: map 75% reduce 0% 15/04/03 03:33:32 INFO mapred.JobClient: map 76% reduce 0% 15/04/03 03:33:49 INFO mapred.JobClient: map 77% reduce 0% 15/04/03 03:34:04 INFO mapred.JobClient: map 78% reduce 0% 15/04/03 03:34:09 INFO mapred.JobClient: map 79% reduce 0% 15/04/03 03:34:16 INFO mapred.JobClient: map 80% reduce 0% 15/04/03 03:34:28 INFO mapred.JobClient: map 81% reduce 0% 15/04/03 03:34:34 INFO mapred.JobClient: map 82% reduce 0% 15/04/03 03:34:43 INFO mapred.JobClient: map 83% reduce 0% 15/04/03 03:34:53 INFO mapred.JobClient: map 84% reduce 0% 15/04/03 03:34:59 INFO mapred.JobClient: map 85% reduce 0% 15/04/03 03:35:08 INFO mapred.JobClient: map 86% reduce 0% 15/04/03 03:35:17 INFO mapred.JobClient: map 87% reduce 0% 15/04/03 03:35:23 INFO mapred.JobClient: map 88% reduce 0% 15/04/03 03:35:32 INFO mapred.JobClient: map 89% reduce 0% 15/04/03 03:35:41 INFO mapred.JobClient: map 90% reduce 0% 15/04/03 03:35:50 INFO mapred.JobClient: map 91% reduce 0% 15/04/03 03:35:53 INFO mapred.JobClient: map 92% reduce 0% 15/04/03 03:36:03 INFO mapred.JobClient: map 93% reduce 0% 15/04/03 03:36:13 INFO mapred.JobClient: map 94% reduce 0% 15/04/03 03:36:25 INFO mapred.JobClient: map 95% reduce 0% 15/04/03 03:36:50 INFO mapred.JobClient: map 96% reduce 0% 15/04/03 03:37:01 INFO mapred.JobClient: map 97% reduce 0% 15/04/03 03:37:46 INFO mapred.JobClient: map 99% reduce 0% 15/04/03 03:38:54 INFO mapred.JobClient: map 100% reduce 0% 15/04/03 03:38:56 INFO mapred.JobClient: Job complete: job_201504022127_0013 15/04/03 03:38:56 INFO mapred.JobClient: Counters: 20 15/04/03 03:38:56 INFO mapred.JobClient: Job Counters 15/04/03 03:38:56 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1839963 15/04/03 03:38:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/03 03:38:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/03 03:38:56 INFO mapred.JobClient: Rack-local map tasks=15 15/04/03 03:38:56 INFO mapred.JobClient: Launched map tasks=60 15/04/03 03:38:56 INFO mapred.JobClient: Data-local map tasks=45 15/04/03 03:38:56 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=6580 15/04/03 03:38:56 INFO mapred.JobClient: File Output Format Counters 15/04/03 03:38:56 INFO mapred.JobClient: Bytes Written=87165324 15/04/03 03:38:56 INFO mapred.JobClient: FileSystemCounters 15/04/03 03:38:56 INFO mapred.JobClient: HDFS_BYTES_READ=181494754 15/04/03 03:38:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2710790 15/04/03 03:38:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=87165324 15/04/03 03:38:56 INFO mapred.JobClient: File Input Format Counters 15/04/03 03:38:56 INFO mapred.JobClient: Bytes Read=181488418 15/04/03 03:38:56 INFO mapred.JobClient: Map-Reduce Framework 15/04/03 03:38:56 INFO mapred.JobClient: Map input records=40000 15/04/03 03:38:56 INFO mapred.JobClient: Physical memory (bytes) snapshot=3853488128 15/04/03 03:38:56 INFO mapred.JobClient: Spilled Records=0 15/04/03 03:38:56 INFO mapred.JobClient: CPU time spent (ms)=134550 15/04/03 03:38:56 INFO mapred.JobClient: Total committed heap usage (bytes)=1525678080 15/04/03 03:38:56 INFO mapred.JobClient: Virtual memory (bytes) snapshot=40598532096 15/04/03 03:38:56 INFO mapred.JobClient: Map output records=40000 15/04/03 03:38:56 INFO mapred.JobClient: SPLIT_RAW_BYTES=6336 15/04/03 03:42:26 INFO input.FileInputFormat: Total input paths to process : 48 15/04/03 03:42:27 INFO mapred.JobClient: Running job: job_201504022127_0014 15/04/03 03:42:28 INFO mapred.JobClient: map 0% reduce 0% 15/04/03 03:42:32 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000049_0, Status : FAILED Error launching task 15/04/03 03:42:34 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000049_1, Status : FAILED Error launching task 15/04/03 03:42:34 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000049_1&filter=stdout 15/04/03 03:42:34 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave0:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000049_1&filter=stderr 15/04/03 03:42:43 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000004_0, Status : FAILED Error launching task 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000004_0&filter=stdout 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000004_0&filter=stderr 15/04/03 03:42:43 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000005_0, Status : FAILED Error launching task 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000005_0&filter=stdout 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000005_0&filter=stderr 15/04/03 03:42:43 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000007_0, Status : FAILED Error launching task 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000007_0&filter=stdout 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000007_0&filter=stderr 15/04/03 03:42:43 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_m_000008_0, Status : FAILED Error launching task 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000008_0&filter=stdout 15/04/03 03:42:43 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_m_000008_0&filter=stderr 15/04/03 03:42:53 INFO mapred.JobClient: map 1% reduce 0% 15/04/03 03:42:55 INFO mapred.JobClient: map 2% reduce 0% 15/04/03 03:43:02 INFO mapred.JobClient: map 3% reduce 0% 15/04/03 03:43:07 INFO mapred.JobClient: map 4% reduce 0% 15/04/03 03:43:14 INFO mapred.JobClient: map 5% reduce 0% 15/04/03 03:43:16 INFO mapred.JobClient: map 6% reduce 0% 15/04/03 03:43:23 INFO mapred.JobClient: map 7% reduce 0% 15/04/03 03:43:30 INFO mapred.JobClient: map 8% reduce 0% 15/04/03 03:43:54 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_r_000002_0, Status : FAILED Error launching task 15/04/03 03:43:54 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000002_0&filter=stdout 15/04/03 03:43:54 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000002_0&filter=stderr 15/04/03 03:43:54 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_r_000006_0, Status : FAILED Error launching task 15/04/03 03:43:54 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000006_0&filter=stdout 15/04/03 03:43:54 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000006_0&filter=stderr 15/04/03 03:44:03 INFO mapred.JobClient: map 9% reduce 0% 15/04/03 03:44:12 INFO mapred.JobClient: map 10% reduce 0% 15/04/03 03:44:16 INFO mapred.JobClient: map 11% reduce 0% 15/04/03 03:44:17 INFO mapred.JobClient: map 12% reduce 0% 15/04/03 03:44:25 INFO mapred.JobClient: map 13% reduce 0% 15/04/03 03:44:29 INFO mapred.JobClient: map 15% reduce 0% 15/04/03 03:44:34 INFO mapred.JobClient: map 16% reduce 0% 15/04/03 03:44:37 INFO mapred.JobClient: map 17% reduce 0% 15/04/03 03:44:41 INFO mapred.JobClient: map 18% reduce 0% 15/04/03 03:44:43 INFO mapred.JobClient: map 19% reduce 0% 15/04/03 03:44:52 INFO mapred.JobClient: map 20% reduce 0% 15/04/03 03:45:11 INFO mapred.JobClient: map 21% reduce 0% 15/04/03 03:45:24 INFO mapred.JobClient: map 22% reduce 0% 15/04/03 03:45:25 INFO mapred.JobClient: map 22% reduce 1% 15/04/03 03:45:26 INFO mapred.JobClient: map 23% reduce 1% 15/04/03 03:45:29 INFO mapred.JobClient: map 24% reduce 1% 15/04/03 03:45:33 INFO mapred.JobClient: map 25% reduce 1% 15/04/03 03:45:38 INFO mapred.JobClient: map 26% reduce 1% 15/04/03 03:45:43 INFO mapred.JobClient: map 27% reduce 1% 15/04/03 03:45:44 INFO mapred.JobClient: map 28% reduce 1% 15/04/03 03:45:50 INFO mapred.JobClient: map 29% reduce 1% 15/04/03 03:45:52 INFO mapred.JobClient: map 30% reduce 1% 15/04/03 03:45:57 INFO mapred.JobClient: map 31% reduce 1% 15/04/03 03:46:04 INFO mapred.JobClient: map 32% reduce 1% 15/04/03 03:46:16 INFO mapred.JobClient: map 33% reduce 1% 15/04/03 03:46:30 INFO mapred.JobClient: map 34% reduce 1% 15/04/03 03:46:36 INFO mapred.JobClient: map 35% reduce 1% 15/04/03 03:46:42 INFO mapred.JobClient: map 35% reduce 2% 15/04/03 03:46:43 INFO mapred.JobClient: map 36% reduce 2% 15/04/03 03:46:48 INFO mapred.JobClient: map 37% reduce 2% 15/04/03 03:46:52 INFO mapred.JobClient: map 38% reduce 2% 15/04/03 03:46:54 INFO mapred.JobClient: map 39% reduce 2% 15/04/03 03:46:55 INFO mapred.JobClient: map 40% reduce 2% 15/04/03 03:46:57 INFO mapred.JobClient: map 41% reduce 2% 15/04/03 03:47:01 INFO mapred.JobClient: map 42% reduce 2% 15/04/03 03:47:04 INFO mapred.JobClient: map 44% reduce 2% 15/04/03 03:47:10 INFO mapred.JobClient: map 45% reduce 2% 15/04/03 03:47:16 INFO mapred.JobClient: map 46% reduce 2% 15/04/03 03:47:17 INFO mapred.JobClient: map 47% reduce 2% 15/04/03 03:47:22 INFO mapred.JobClient: map 48% reduce 2% 15/04/03 03:47:29 INFO mapred.JobClient: map 49% reduce 2% 15/04/03 03:47:36 INFO mapred.JobClient: map 50% reduce 2% 15/04/03 03:47:40 INFO mapred.JobClient: map 51% reduce 2% 15/04/03 03:47:47 INFO mapred.JobClient: map 51% reduce 3% 15/04/03 03:47:52 INFO mapred.JobClient: map 52% reduce 3% 15/04/03 03:47:59 INFO mapred.JobClient: map 53% reduce 3% 15/04/03 03:48:02 INFO mapred.JobClient: map 54% reduce 3% 15/04/03 03:48:08 INFO mapred.JobClient: map 55% reduce 3% 15/04/03 03:48:09 INFO mapred.JobClient: map 56% reduce 3% 15/04/03 03:48:11 INFO mapred.JobClient: map 57% reduce 3% 15/04/03 03:48:13 INFO mapred.JobClient: map 58% reduce 3% 15/04/03 03:48:20 INFO mapred.JobClient: map 59% reduce 3% 15/04/03 03:48:24 INFO mapred.JobClient: map 60% reduce 3% 15/04/03 03:48:33 INFO mapred.JobClient: map 61% reduce 3% 15/04/03 03:48:35 INFO mapred.JobClient: map 62% reduce 3% 15/04/03 03:48:43 INFO mapred.JobClient: map 63% reduce 3% 15/04/03 03:48:44 INFO mapred.JobClient: map 63% reduce 4% 15/04/03 03:48:51 INFO mapred.JobClient: map 65% reduce 4% 15/04/03 03:49:03 INFO mapred.JobClient: map 66% reduce 4% 15/04/03 03:49:04 INFO mapred.JobClient: map 67% reduce 4% 15/04/03 03:49:12 INFO mapred.JobClient: map 68% reduce 4% 15/04/03 03:49:15 INFO mapred.JobClient: map 70% reduce 4% 15/04/03 03:49:24 INFO mapred.JobClient: map 71% reduce 4% 15/04/03 03:49:26 INFO mapred.JobClient: map 72% reduce 4% 15/04/03 03:49:34 INFO mapred.JobClient: map 73% reduce 4% 15/04/03 03:49:38 INFO mapred.JobClient: map 74% reduce 5% 15/04/03 03:49:40 INFO mapred.JobClient: map 75% reduce 5% 15/04/03 03:49:48 INFO mapred.JobClient: map 76% reduce 5% 15/04/03 03:49:49 INFO mapred.JobClient: map 77% reduce 5% 15/04/03 03:49:51 INFO mapred.JobClient: map 78% reduce 5% 15/04/03 03:49:59 INFO mapred.JobClient: map 79% reduce 5% 15/04/03 03:50:00 INFO mapred.JobClient: map 80% reduce 5% 15/04/03 03:50:03 INFO mapred.JobClient: map 81% reduce 5% 15/04/03 03:50:15 INFO mapred.JobClient: map 83% reduce 5% 15/04/03 03:50:25 INFO mapred.JobClient: map 84% reduce 5% 15/04/03 03:50:27 INFO mapred.JobClient: map 85% reduce 5% 15/04/03 03:50:28 INFO mapred.JobClient: map 86% reduce 5% 15/04/03 03:50:36 INFO mapred.JobClient: map 87% reduce 5% 15/04/03 03:50:38 INFO mapred.JobClient: map 88% reduce 5% 15/04/03 03:50:46 INFO mapred.JobClient: map 89% reduce 5% 15/04/03 03:50:47 INFO mapred.JobClient: map 90% reduce 5% 15/04/03 03:50:50 INFO mapred.JobClient: map 91% reduce 5% 15/04/03 03:50:53 INFO mapred.JobClient: map 92% reduce 5% 15/04/03 03:50:54 INFO mapred.JobClient: map 92% reduce 6% 15/04/03 03:51:00 INFO mapred.JobClient: map 93% reduce 6% 15/04/03 03:51:06 INFO mapred.JobClient: map 94% reduce 6% 15/04/03 03:51:09 INFO mapred.JobClient: map 95% reduce 6% 15/04/03 03:51:14 INFO mapred.JobClient: map 96% reduce 6% 15/04/03 03:51:28 INFO mapred.JobClient: map 97% reduce 6% 15/04/03 03:51:29 INFO mapred.JobClient: map 98% reduce 6% 15/04/03 03:51:31 INFO mapred.JobClient: map 98% reduce 7% 15/04/03 03:51:47 INFO mapred.JobClient: map 99% reduce 7% 15/04/03 03:51:59 INFO mapred.JobClient: map 100% reduce 7% 15/04/03 03:52:59 INFO mapred.JobClient: map 100% reduce 8% 15/04/03 03:53:02 INFO mapred.JobClient: map 100% reduce 9% 15/04/03 03:53:05 INFO mapred.JobClient: map 100% reduce 10% 15/04/03 03:53:08 INFO mapred.JobClient: map 100% reduce 12% 15/04/03 03:53:11 INFO mapred.JobClient: map 100% reduce 13% 15/04/03 03:53:20 INFO mapred.JobClient: map 100% reduce 15% 15/04/03 03:53:23 INFO mapred.JobClient: map 100% reduce 16% 15/04/03 03:53:46 INFO mapred.JobClient: map 100% reduce 17% 15/04/03 03:53:50 INFO mapred.JobClient: map 100% reduce 20% 15/04/03 03:53:53 INFO mapred.JobClient: map 100% reduce 22% 15/04/03 03:53:56 INFO mapred.JobClient: map 100% reduce 24% 15/04/03 03:53:59 INFO mapred.JobClient: map 100% reduce 25% 15/04/03 03:54:01 INFO mapred.JobClient: map 100% reduce 26% 15/04/03 03:54:13 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_r_000011_0, Status : FAILED Error launching task 15/04/03 03:54:13 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000011_0&filter=stdout 15/04/03 03:54:13 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000011_0&filter=stderr 15/04/03 03:54:15 INFO mapred.JobClient: Task Id : attempt_201504022127_0014_r_000012_0, Status : FAILED Error launching task 15/04/03 03:54:15 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000012_0&filter=stdout 15/04/03 03:54:15 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0014_r_000012_0&filter=stderr 15/04/03 03:54:28 INFO mapred.JobClient: map 100% reduce 27% 15/04/03 03:54:41 INFO mapred.JobClient: map 100% reduce 28% 15/04/03 03:55:10 INFO mapred.JobClient: map 100% reduce 29% 15/04/03 03:55:44 INFO mapred.JobClient: map 100% reduce 30% 15/04/03 03:56:20 INFO mapred.JobClient: map 100% reduce 31% 15/04/03 03:56:47 INFO mapred.JobClient: map 100% reduce 32% 15/04/03 03:56:50 INFO mapred.JobClient: map 100% reduce 33% 15/04/03 03:56:53 INFO mapred.JobClient: map 100% reduce 34% 15/04/03 03:57:17 INFO mapred.JobClient: map 100% reduce 35% 15/04/03 03:58:08 INFO mapred.JobClient: map 100% reduce 36% 15/04/03 03:58:35 INFO mapred.JobClient: map 100% reduce 38% 15/04/03 03:58:41 INFO mapred.JobClient: map 100% reduce 39% 15/04/03 03:58:51 INFO mapred.JobClient: map 100% reduce 40% 15/04/03 03:58:54 INFO mapred.JobClient: map 100% reduce 41% 15/04/03 03:58:56 INFO mapred.JobClient: map 100% reduce 42% 15/04/03 03:59:02 INFO mapred.JobClient: map 100% reduce 43% 15/04/03 03:59:05 INFO mapred.JobClient: map 100% reduce 44% 15/04/03 03:59:08 INFO mapred.JobClient: map 100% reduce 45% 15/04/03 03:59:40 INFO mapred.JobClient: map 100% reduce 46% 15/04/03 04:00:07 INFO mapred.JobClient: map 100% reduce 47% 15/04/03 04:00:25 INFO mapred.JobClient: map 100% reduce 49% 15/04/03 04:00:28 INFO mapred.JobClient: map 100% reduce 50% 15/04/03 04:01:07 INFO mapred.JobClient: map 100% reduce 51% 15/04/03 04:01:20 INFO mapred.JobClient: map 100% reduce 52% 15/04/03 04:01:21 INFO mapred.JobClient: map 100% reduce 53% 15/04/03 04:01:26 INFO mapred.JobClient: map 100% reduce 54% 15/04/03 04:02:01 INFO mapred.JobClient: map 100% reduce 56% 15/04/03 04:02:04 INFO mapred.JobClient: map 100% reduce 57% 15/04/03 04:02:19 INFO mapred.JobClient: map 100% reduce 58% 15/04/03 04:02:54 INFO mapred.JobClient: map 100% reduce 59% 15/04/03 04:03:18 INFO mapred.JobClient: map 100% reduce 61% 15/04/03 04:03:24 INFO mapred.JobClient: map 100% reduce 62% 15/04/03 04:03:42 INFO mapred.JobClient: map 100% reduce 63% 15/04/03 04:04:30 INFO mapred.JobClient: map 100% reduce 64% 15/04/03 04:05:28 INFO mapred.JobClient: map 100% reduce 65% 15/04/03 04:05:51 INFO mapred.JobClient: map 100% reduce 67% 15/04/03 04:05:54 INFO mapred.JobClient: map 100% reduce 68% 15/04/03 04:06:43 INFO mapred.JobClient: map 100% reduce 69% 15/04/03 04:07:16 INFO mapred.JobClient: map 100% reduce 71% 15/04/03 04:07:32 INFO mapred.JobClient: map 100% reduce 73% 15/04/03 04:07:34 INFO mapred.JobClient: map 100% reduce 74% 15/04/03 04:07:35 INFO mapred.JobClient: map 100% reduce 75% 15/04/03 04:08:04 INFO mapred.JobClient: map 100% reduce 77% 15/04/03 04:08:07 INFO mapred.JobClient: map 100% reduce 78% 15/04/03 04:08:24 INFO mapred.JobClient: map 100% reduce 79% 15/04/03 04:09:01 INFO mapred.JobClient: map 100% reduce 80% 15/04/03 04:09:18 INFO mapred.JobClient: map 100% reduce 82% 15/04/03 04:09:22 INFO mapred.JobClient: map 100% reduce 83% 15/04/03 04:09:50 INFO mapred.JobClient: map 100% reduce 84% 15/04/03 04:10:03 INFO mapred.JobClient: map 100% reduce 85% 15/04/03 04:10:11 INFO mapred.JobClient: map 100% reduce 86% 15/04/03 04:10:35 INFO mapred.JobClient: map 100% reduce 87% 15/04/03 04:11:32 INFO mapred.JobClient: map 100% reduce 88% 15/04/03 04:11:46 INFO mapred.JobClient: map 100% reduce 90% 15/04/03 04:11:59 INFO mapred.JobClient: map 100% reduce 91% 15/04/03 04:12:32 INFO mapred.JobClient: map 100% reduce 93% 15/04/03 04:12:35 INFO mapred.JobClient: map 100% reduce 94% 15/04/03 04:13:30 INFO mapred.JobClient: map 100% reduce 96% 15/04/03 04:13:33 INFO mapred.JobClient: map 100% reduce 97% 15/04/03 04:13:44 INFO mapred.JobClient: map 100% reduce 98% 15/04/03 04:13:47 INFO mapred.JobClient: map 100% reduce 99% 15/04/03 04:14:05 INFO mapred.JobClient: map 100% reduce 100% 15/04/03 04:14:14 INFO mapred.JobClient: Job complete: job_201504022127_0014 15/04/03 04:14:14 INFO mapred.JobClient: Counters: 32 15/04/03 04:14:14 INFO mapred.JobClient: Job Counters 15/04/03 04:14:14 INFO mapred.JobClient: Launched reduce tasks=31 15/04/03 04:14:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=3712107 15/04/03 04:14:14 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/03 04:14:14 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/03 04:14:14 INFO mapred.JobClient: Rack-local map tasks=3 15/04/03 04:14:14 INFO mapred.JobClient: Launched map tasks=54 15/04/03 04:14:14 INFO mapred.JobClient: Data-local map tasks=51 15/04/03 04:14:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10240649 15/04/03 04:14:14 INFO mapred.JobClient: org.apache.mahout.vectorizer.collocations.llr.CollocMapper$Count 15/04/03 04:14:14 INFO mapred.JobClient: NGRAM_TOTAL=44741934 15/04/03 04:14:14 INFO mapred.JobClient: File Output Format Counters 15/04/03 04:14:14 INFO mapred.JobClient: Bytes Written=26509282 15/04/03 04:14:14 INFO mapred.JobClient: org.apache.mahout.vectorizer.collocations.llr.CollocReducer$Skipped 15/04/03 04:14:14 INFO mapred.JobClient: LESS_THAN_MIN_SUPPORT=68210076 15/04/03 04:14:14 INFO mapred.JobClient: FileSystemCounters 15/04/03 04:14:14 INFO mapred.JobClient: FILE_BYTES_READ=15324597526 15/04/03 04:14:14 INFO mapred.JobClient: HDFS_BYTES_READ=87173148 15/04/03 04:14:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=21404924440 15/04/03 04:14:14 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=26509282 15/04/03 04:14:14 INFO mapred.JobClient: File Input Format Counters 15/04/03 04:14:14 INFO mapred.JobClient: Bytes Read=87165324 15/04/03 04:14:14 INFO mapred.JobClient: Map-Reduce Framework 15/04/03 04:14:14 INFO mapred.JobClient: Map output materialized bytes=6112297187 15/04/03 04:14:14 INFO mapred.JobClient: Map input records=40000 15/04/03 04:14:14 INFO mapred.JobClient: Reduce shuffle bytes=6112297187 15/04/03 04:14:14 INFO mapred.JobClient: Spilled Records=424321293 15/04/03 04:14:14 INFO mapred.JobClient: Map output bytes=7393388643 15/04/03 04:14:14 INFO mapred.JobClient: Total committed heap usage (bytes)=11110109184 15/04/03 04:14:14 INFO mapred.JobClient: CPU time spent (ms)=1960340 15/04/03 04:14:14 INFO mapred.JobClient: Combine input records=422967584 15/04/03 04:14:14 INFO mapred.JobClient: SPLIT_RAW_BYTES=7824 15/04/03 04:14:14 INFO mapred.JobClient: Reduce input records=100254078 15/04/03 04:14:14 INFO mapred.JobClient: Reduce input groups=16732774 15/04/03 04:14:14 INFO mapred.JobClient: Combine output records=331241009 15/04/03 04:14:14 INFO mapred.JobClient: Physical memory (bytes) snapshot=14863052800 15/04/03 04:14:14 INFO mapred.JobClient: Reduce output records=3504184 15/04/03 04:14:14 INFO mapred.JobClient: Virtual memory (bytes) snapshot=61045489664 15/04/03 04:14:14 INFO mapred.JobClient: Map output records=191980653 15/04/03 04:18:42 INFO input.FileInputFormat: Total input paths to process : 24 15/04/03 04:18:43 INFO mapred.JobClient: Running job: job_201504022127_0015 15/04/03 04:18:44 INFO mapred.JobClient: map 0% reduce 0% 15/04/03 04:18:48 INFO mapred.JobClient: Task Id : attempt_201504022127_0015_m_000025_0, Status : FAILED Error launching task 15/04/03 04:18:48 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_m_000025_0&filter=stdout 15/04/03 04:18:48 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_m_000025_0&filter=stderr 15/04/03 04:18:50 INFO mapred.JobClient: Task Id : attempt_201504022127_0015_m_000025_1, Status : FAILED Error launching task 15/04/03 04:18:50 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_m_000025_1&filter=stdout 15/04/03 04:18:50 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_m_000025_1&filter=stderr 15/04/03 04:18:53 INFO mapred.JobClient: Task Id : attempt_201504022127_0015_r_000025_0, Status : FAILED Error launching task 15/04/03 04:18:53 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_r_000025_0&filter=stdout 15/04/03 04:18:53 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504022127_0015_r_000025_0&filter=stderr 15/04/03 04:18:56 INFO mapred.JobClient: Task Id : attempt_201504022127_0015_m_000025_2, Status : FAILED Error launching task 15/04/03 04:19:02 INFO mapred.JobClient: Job complete: job_201504022127_0015 15/04/03 04:19:02 INFO mapred.JobClient: Counters: 4 15/04/03 04:19:02 INFO mapred.JobClient: Job Counters 15/04/03 04:19:02 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=3263 15/04/03 04:19:02 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/03 04:19:02 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/03 04:19:02 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 Exception in thread "main" java.lang.IllegalStateException: Job failed! at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.computeNGramsPruneByLLR(CollocDriver.java:281) at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateAllGrams(CollocDriver.java:191) at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:183) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:271) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:55) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Warning: $HADOOP_HOME is deprecated.

Running on hadoop, using /home/fedora/hadoop-1.2.1/bin/hadoop and HADOOP_CONF_DIR=/home/fedora/hadoop-1.2.1/conf MAHOUT-JOB: /home/fedora/HiBench/common/hibench/mahout/target/mahout-distribution-0.7/mahout-examples-0.7-job.jar Warning: $HADOOP_HOME is deprecated.

15/04/03 04:19:07 WARN driver.MahoutDriver: No trainnb.props found on classpath, will use command-line arguments only 15/04/03 04:19:07 INFO common.AbstractJob: Command line arguments: {--alphaI=[1.0], --endPhase=[2147483647], --extractLabels=null, --input=[/home/fedora/Bayes/Output-comp/vectors/tfidf-vectors], --labelIndex=[/home/fedora/Bayes/Output-comp/labelindex], --output=[/home/fedora/Bayes/Output-comp/model], --overwrite=null, --startPhase=[0], --tempDir=[/home/fedora/Bayes/Output-comp/temp]} 15/04/03 04:23:55 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.111.240:9000/tmp/hadoop-fedora/mapred/staging/fedora/.staging/job_201504022127_0016 15/04/03 04:23:55 ERROR security.UserGroupInformation: PriviledgedActionException as:fedora cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235) at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:55) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071) at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) at org.apache.mahout.classifier.naivebayes.training.TrainNaiveBayesJob.run(TrainNaiveBayesJob.java:105) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.mahout.classifier.naivebayes.training.TrainNaiveBayesJob.main(TrainNaiveBayesJob.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

(The following are the main command in the run.sh that are executed) /home/fedora/HiBench/bin/../common/hibench/mahout/target/mahout-distribution-0.7/bin/mahout seq2sparse -Dmapred.map.output.compress=true -Dmapred.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compression.type=BLOCK -i /home/fedora/Bayes/Input-comp -o /home/fedora/Bayes/Output-comp/vectors -lnorm -nv -wt tfidf -ng 3 --numReducers 24 /home/fedora/HiBench/bin/../common/hibench/mahout/target/mahout-distribution-0.7/bin/mahout trainnb -Dmapred.map.output.compress=true -Dmapred.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compression.type=BLOCK -i /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors -el -o /home/fedora/Bayes/Output-comp/model -li /home/fedora/Bayes/Output-comp/labelindex -ow --tempDir /home/fedora/Bayes/Output-comp/temp

adrian-wang commented 9 years ago

if you are using hadoop 1.2.1, then hadoop version should be set as hadoop1 instead of hadoop2. export HADOOP_VERSION=hadoop1

echozyw commented 9 years ago

Thanks for your insight. But that is already set via the following statement: export HADOOP_VERSION=hadoop2 # set it to hadoop1 to enable MR1, hadoop2 to enable MR2

if $HADOOP_EXECUTABLE version|grep -i -q cdh4; then HADOOP_RELEASE=cdh4 HADOOP_VERSION=cdh4 elif $HADOOP_EXECUTABLE version|grep -i -q cdh5; then HADOOP_RELEASE=cdh5 HADOOP_VERSION=cdh5 elif $HADOOP_EXECUTABLE version|grep -i -q "hadoop 2"; then HADOOP_RELEASE=hadoop2 HADOOP_VERSION=hadoop2 else HADOOP_RELEASE=hadoop1 HADOOP_VERSION=hadoop1 (@@@@@@This will automatically set the Hadoop version to hadoop1) fi

Any other speculations? I am kindof stuck here. Any input will be greatly appreciated!

Thanks a lot, Gina

adrian-wang commented 9 years ago

Can you get the log of tasktracker?

echozyw commented 9 years ago

Here is some high-level description: I have a 5-node vm-based cluster, including one master and 4 data/compute nodes. For the bayes workload, the following error shows up at the same place during ./run.sh. I checked all the 4 slave nodes (logs/tasktracker*.log and logs/userlogs/JOBFOLDER) and I found that only one slave has job logs (in logs/userlogs/JOBFOLDER) with "stderr" information which I also pasted below, along with the tasktracker.log of the same slave node. Thanks a lot in advance! !!!!!!0) The Error output while performing: ./run.sh 15/04/07 03:15:03 INFO mapred.JobClient: map 84% reduce 0% 15/04/07 03:15:18 INFO mapred.JobClient: map 85% reduce 0% 15/04/07 03:15:20 INFO mapred.JobClient: map 86% reduce 0% 15/04/07 03:15:21 INFO mapred.JobClient: map 87% reduce 0% 15/04/07 03:15:39 INFO mapred.JobClient: map 88% reduce 0% 15/04/07 03:15:43 INFO mapred.JobClient: map 89% reduce 0% 15/04/07 03:15:51 INFO mapred.JobClient: map 90% reduce 0% 15/04/07 03:16:01 INFO mapred.JobClient: map 91% reduce 0% 15/04/07 03:16:16 INFO mapred.JobClient: map 93% reduce 0% 15/04/07 03:16:22 INFO mapred.JobClient: map 94% reduce 0% 15/04/07 03:16:31 INFO mapred.JobClient: map 95% reduce 0% 15/04/07 03:17:14 INFO mapred.JobClient: map 97% reduce 0% 15/04/07 03:17:30 INFO mapred.JobClient: map 99% reduce 0% 15/04/07 03:17:53 INFO mapred.JobClient: map 100% reduce 0% 15/04/07 03:17:55 INFO mapred.JobClient: Job complete: job_201504031414_0032 15/04/07 03:17:55 INFO mapred.JobClient: Counters: 20 15/04/07 03:17:55 INFO mapred.JobClient: Job Counters 15/04/07 03:17:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1467609 15/04/07 03:17:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/07 03:17:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/07 03:17:55 INFO mapred.JobClient: Rack-local map tasks=13 15/04/07 03:17:55 INFO mapred.JobClient: Launched map tasks=60 15/04/07 03:17:55 INFO mapred.JobClient: Data-local map tasks=47 15/04/07 03:17:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=5694 15/04/07 03:17:55 INFO mapred.JobClient: File Output Format Counters 15/04/07 03:17:55 INFO mapred.JobClient: Bytes Written=87165324 15/04/07 03:17:55 INFO mapred.JobClient: FileSystemCounters 15/04/07 03:17:55 INFO mapred.JobClient: HDFS_BYTES_READ=181494754 15/04/07 03:17:55 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2710790 15/04/07 03:17:55 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=87165324 15/04/07 03:17:55 INFO mapred.JobClient: File Input Format Counters 15/04/07 03:17:55 INFO mapred.JobClient: Bytes Read=181488418 15/04/07 03:17:55 INFO mapred.JobClient: Map-Reduce Framework 15/04/07 03:17:55 INFO mapred.JobClient: Map input records=40000 15/04/07 03:17:55 INFO mapred.JobClient: Physical memory (bytes) snapshot=3825807360 15/04/07 03:17:55 INFO mapred.JobClient: Spilled Records=0 15/04/07 03:17:55 INFO mapred.JobClient: CPU time spent (ms)=117680 15/04/07 03:17:55 INFO mapred.JobClient: Total committed heap usage (bytes)=1525678080 15/04/07 03:17:55 INFO mapred.JobClient: Virtual memory (bytes) snapshot=40598466560 15/04/07 03:17:55 INFO mapred.JobClient: Map output records=40000 15/04/07 03:17:55 INFO mapred.JobClient: SPLIT_RAW_BYTES=6336

15/04/07 03:21:59 INFO input.FileInputFormat: Total input paths to process : 48 15/04/07 03:22:00 INFO mapred.JobClient: Running job: job_201504031414_0033 15/04/07 03:22:01 INFO mapred.JobClient: map 0% reduce 0% 15/04/07 03:22:04 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_0, Status : FAILED Error launching task 15/04/07 03:22:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_0&filter=stdout 15/04/07 03:22:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_0&filter=stderr 15/04/07 03:22:07 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_1, Status : FAILED Error launching task 15/04/07 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_1&filter=stdout 15/04/07 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_1&filter=stderr 15/04/07 03:22:09 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_r_000025_0, Status : FAILED Error launching task 15/04/07 03:22:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_r_000025_0&filter=stdout 15/04/07 03:22:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_r_000025_0&filter=stderr 15/04/07 03:22:12 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_2, Status : FAILED Error launching task 15/04/07 03:22:12 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_2&filter=stdout 15/04/07 03:22:12 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_2&filter=stderr 15/04/07 03:22:21 INFO mapred.JobClient: Job complete: job_201504031414_0033 15/04/07 03:22:21 INFO mapred.JobClient: Counters: 4 15/04/07 03:22:21 INFO mapred.JobClient: Job Counters 15/04/07 03:22:21 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=5062 15/04/07 03:22:21 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/07 03:22:21 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/07 03:22:21 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 Exception in thread "main" java.lang.IllegalStateException: Job failed! at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateCollocations(CollocDriver.java:239) at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateAllGrams(CollocDriver.java:188) at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:183) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:271) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:55) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Warning: $HADOOP_HOME is deprecated.

Running on hadoop, using /home/fedora/hadoop-1.2.1/bin/hadoop and HADOOP_CONF_DIR=/home/fedora/hadoop-1.2.1/conf MAHOUT-JOB: /home/fedora/HiBench/common/hibench/mahout/target/mahout-distribution-0.7/mahout-examples-0.7-job.jar Warning: $HADOOP_HOME is deprecated.

15/04/07 03:22:27 WARN driver.MahoutDriver: No trainnb.props found on classpath, will use command-line arguments only 15/04/07 03:22:27 INFO common.AbstractJob: Command line arguments: {--alphaI=[1.0], --endPhase=[2147483647], --extractLabels=null, --input=[/home/fedora/Bayes/Output-comp/vectors/tfidf-vectors], --labelIndex=[/home/fedora/Bayes/Output-comp/labelindex], --output=[/home/fedora/Bayes/Output-comp/model], --overwrite=null, --startPhase=[0], --tempDir=[/home/fedora/Bayes/Output-comp/temp]} 15/04/07 03:26:33 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.111.240:9000/tmp/hadoop-fedora/mapred/staging/fedora/.staging/job_201504031414_0034 15/04/07 03:26:33 ERROR security.UserGroupInformation: PriviledgedActionException as:fedora cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235) at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:55) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071) at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) at org.apache.mahout.classifier.naivebayes.training.TrainNaiveBayesJob.run(TrainNaiveBayesJob.java:105) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.mahout.classifier.naivebayes.training.TrainNaiveBayesJob.main(TrainNaiveBayesJob.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

/home/fedora/HiBench/bin/../common/hibench/mahout/target/mahout-distribution-0.7/bin/mahout seq2sparse -Dmapred.map.output.compress=true -Dmapred.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compression.type=BLOCK -i /home/fedora/Bayes/Input-comp -o /home/fedora/Bayes/Output-comp/vectors -lnorm -nv -wt tfidf -ng 3 --numReducers 24 /home/fedora/HiBench/bin/../common/hibench/mahout/target/mahout-distribution-0.7/bin/mahout trainnb -Dmapred.map.output.compress=true -Dmapred.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec -Dmapred.output.compression.type=BLOCK -i /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors -el -o /home/fedora/Bayes/Output-comp/model -li /home/fedora/Bayes/Output-comp/labelindex -ow --tempDir /home/fedora/Bayes/Output-comp/temp Warning: $HADOOP_HOME is deprecated.

!!!!!!1) userlogs/JOBID/attemptID 1 log4j:WARN Failed to set property [conversionPattern] to value "%d{ISO8601} %p %c: %m%n". 2 java.lang.reflect.InvocationTargetException 3 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 5 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 6 at java.lang.reflect.Method.invoke(Method.java:606) 7 at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:206) 8 at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:165) 9 at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:130) 10 at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97) 11 at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:684) 12 at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647) 13 at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544) 14 at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440) 15 at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476) 16 at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471) 17 at org.apache.log4j.LogManager.(LogManager.java:125) 18 at org.apache.log4j.Logger.getLogger(Logger.java:105) 19 at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:289) 20 at org.apache.commons.logging.impl.Log4JLogger.(Log4JLogger.java:109) 21 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 22 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 23 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 24 at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 25 at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1116) 26 at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:914) 27 at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604) 28 at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336) 29 at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310) 30 at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) 31 at org.apache.hadoop.mapred.Child.(Child.java:56) 32 Caused by: java.lang.ExceptionInInitializerError 33 at java.nio.file.FileSystems.getDefault(FileSystems.java:176) 34 at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:490) 35 at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:481) 36 at java.security.AccessController.doPrivileged(Native Method) 37 at sun.util.calendar.ZoneInfoFile.(ZoneInfoFile.java:480) 38 at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:663) 39 at java.util.TimeZone.getTimeZone(TimeZone.java:566) 40 at java.util.TimeZone.setDefaultZone(TimeZone.java:663) 41 at java.util.TimeZone.getDefaultRef(TimeZone.java:630) 42 at java.util.Calendar.getInstance(Calendar.java:968) 43 at org.apache.log4j.helpers.AbsoluteTimeDateFormat.(AbsoluteTimeDateFormat.java:62) 44 at org.apache.log4j.helpers.ISO8601DateFormat.(ISO8601DateFormat.java:46) 45 at org.apache.log4j.helpers.PatternParser.finalizeConverter(PatternParser.java:257) 46 at org.apache.log4j.helpers.PatternParser.parse(PatternParser.java:187) 47 at org.apache.log4j.PatternLayout.setConversionPattern(PatternLayout.java:446) 48 ... 29 more 49 Caused by: java.security.PrivilegedActionException: java.security.PrivilegedActionException: sun.nio.fs.UnixException: No such file or directory 50 at java.security.AccessController.doPrivileged(Native Method) 51 at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:95) 52 at java.nio.file.FileSystems$DefaultFileSystemHolder.(FileSystems.java:90) 53 ... 44 more 54 Caused by: java.security.PrivilegedActionException: sun.nio.fs.UnixException: No such file or directory 55 at java.security.AccessController.doPrivileged(Native Method) 56 at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:42) 57 at sun.nio.fs.DefaultFileSystemProvider.create(DefaultFileSystemProvider.java:70) 58 at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:108) 59 at java.nio.file.FileSystems$DefaultFileSystemHolder.access$000(FileSystems.java:89) 60 at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:98) 61 at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:96) 62 ... 47 more 63 Caused by: sun.nio.fs.UnixException: No such file or directory 64 at sun.nio.fs.UnixNativeDispatcher.getcwd(Native Method) 65 at sun.nio.fs.UnixFileSystem.(UnixFileSystem.java:67) 66 at sun.nio.fs.LinuxFileSystem.(LinuxFileSystem.java:39) 67 at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:44) 68 at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:37) 69 at sun.nio.fs.UnixFileSystemProvider.(UnixFileSystemProvider.java:56) 70 at sun.nio.fs.LinuxFileSystemProvider.(LinuxFileSystemProvider.java:39) 71 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 72 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 73 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 74 at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 75 at java.lang.Class.newInstance(Class.java:379) 76 at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:52) 77 at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:43) 78 ... 54 more 79 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/xerces/xinclude/XIncludeHandler 80 at org.apache.xerces.parsers.XIncludeAwareParserConfiguration.configurePipeline(Unknown Source) 81 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) 82 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) 83 at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) 84 at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) 85 at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) 86 at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177) 87 at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156) 88 at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107) 89 at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053) 90 at org.apache.hadoop.conf.Configuration.get(Configuration.java:397) 91 at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:1899) 92 at org.apache.hadoop.mapred.JobConf.(JobConf.java:343) 93 at org.apache.hadoop.mapred.Child.main(Child.java:72) 94 Caused by: java.lang.ClassNotFoundException: org.apache.xerces.xinclude.XIncludeHandler 95 at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 96 at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 97 at java.security.AccessController.doPrivileged(Native Method) 98 at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 99 at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 100 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 101 at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 102 ... 14 more

!!!!!!2) logs/hadoop-fedora-tasktracker-newjobtestslave3.log 1 2015-04-07 03:04:57,360 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0032_m_000003_0 task's state:UNASSIGNED 2 2015-04-07 03:04:57,361 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0032_m_000003_0 which needs 1 slots 3 2015-04-07 03:04:57,362 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201504031414_0032_m_000003_0 which needs 1 slots 4 2015-04-07 03:04:57,364 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0032_m_000006_0 task's state:UNASSIGNED 5 2015-04-07 03:04:57,364 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0032_m_000006_0 which needs 1 slots 6 2015-04-07 03:04:57,365 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201504031414_0032_m_000006_0 which needs 1 slots 7 2015-04-07 03:04:57,432 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user fedora on this TT. 8 2015-04-07 03:05:02,107 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201504031414_0032m-1006288762 9 2015-04-07 03:05:02,107 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201504031414_0032m-1006288762 spawned. 10 2015-04-07 03:05:02,109 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201504031414_0032m-1207767792 11 2015-04-07 03:05:02,111 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201504031414_0032m-1207767792 spawned. 12 2015-04-07 03:05:02,114 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-fedora/mapred/local/ttprivate/taskTracker/fedora/jobcache/job_201504 031414_0032/attempt_201504031414_0032_m_000006_0/taskjvm.sh 13 2015-04-07 03:05:02,116 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-fedora/mapred/local/ttprivate/taskTracker/fedora/jobcache/job_201504 031414_0032/attempt_201504031414_0032_m_000003_0/taskjvm.sh 14 2015-04-07 03:05:05,855 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201504031414_0032m-1207767792 given task: attempt_201504031414_0032_m_000003_0 15 2015-04-07 03:05:05,866 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201504031414_0032m-1006288762 given task: attempt_201504031414_0032_m_000006_0 16 2015-04-07 03:05:14,233 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 0.53208405% 17 2015-04-07 03:05:14,258 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 0.8006324% 18 2015-04-07 03:05:17,261 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 0.53208405% 19 2015-04-07 03:05:17,282 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 20 2015-04-07 03:05:18,084 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201504031414_0032_m_000006_0 is in commit-pending, task state:COMMIT_PENDING 21 2015-04-07 03:05:18,084 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 22 2015-04-07 03:05:18,277 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0032_m_000004_1 task's state:UNASSIGNED 23 2015-04-07 03:05:18,277 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0032_m_000004_1 which needs 1 slots 24 2015-04-07 03:05:18,277 INFO org.apache.hadoop.mapred.TaskTracker: TaskLauncher : Waiting for 1 to launch attempt_201504031414_0032_m_000004_1, currently we have 0 free s lots 25 2015-04-07 03:05:20,325 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 26 2015-04-07 03:05:23,284 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 27 2015-04-07 03:05:23,344 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 28 2015-04-07 03:05:26,328 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 29 2015-04-07 03:05:26,367 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 30 2015-04-07 03:05:29,347 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 31 2015-04-07 03:05:29,385 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 32 2015-04-07 03:05:31,929 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201504031414_0032_m_000003_0 is in commit-pending, task state:COMMIT_PENDING 33 2015-04-07 03:05:31,929 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 34 2015-04-07 03:05:32,220 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0032_m_000005_1 task's state:UNASSIGNED 35 2015-04-07 03:05:32,402 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 36 2015-04-07 03:05:35,485 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 37 2015-04-07 03:05:38,369 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 38 2015-04-07 03:05:38,504 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 39 2015-04-07 03:05:41,394 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 40 2015-04-07 03:05:41,524 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 41 2015-04-07 03:05:44,411 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 42 2015-04-07 03:05:44,541 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% ............. 523 2015-04-07 03:17:51,318 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 524 2015-04-07 03:17:51,477 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 525 2015-04-07 03:17:54,327 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000006_0 1.0% 526 2015-04-07 03:17:54,487 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201504031414_0032_m_000003_0 1.0% 527 2015-04-07 03:17:55,850 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201504031414_0032 528 2015-04-07 03:17:55,857 INFO org.apache.hadoop.util.ProcessTree: Killing process group17828 with signal TERM. Exit code 0 529 2015-04-07 03:17:55,857 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1 530 2015-04-07 03:17:55,858 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201504031414_0032_m_000003_0 not found in cache 531 2015-04-07 03:17:55,858 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201504031414_0032_m_000004_1 which needs 1 slots 532 2015-04-07 03:17:55,862 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0032_m_000005_1 which needs 1 slots 533 2015-04-07 03:17:55,862 INFO org.apache.hadoop.mapred.TaskTracker: TaskLauncher : Waiting for 1 to launch attempt_201504031414_0032_m_000005_1, currently we have 0 free s lots 534 2015-04-07 03:17:55,872 INFO org.apache.hadoop.util.ProcessTree: Killing process group17827 with signal TERM. Exit code 0 535 2015-04-07 03:17:55,872 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1 536 2015-04-07 03:17:55,872 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201504031414_0032_m_000006_0 not found in cache 537 2015-04-07 03:17:55,873 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201504031414_0032_m_000005_1 which needs 1 slots 538 2015-04-07 03:17:55,878 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143 539 2015-04-07 03:17:55,878 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 540 2015-04-07 03:17:55,878 INFO org.apache.hadoop.mapred.TaskController: 541 2015-04-07 03:17:55,878 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201504031414_0032m-1207767792 exited with exit code 143. Number of tasks it ran: 0 542 2015-04-07 03:17:55,879 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201504031414_0032 for user-log deletion with retainTimeStamp:1428463075878 543 2015-04-07 03:17:55,927 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName fedora for UID 1000 from the native implementation 544 2015-04-07 03:17:55,933 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143 545 2015-04-07 03:17:55,933 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 546 2015-04-07 03:17:55,933 INFO org.apache.hadoop.mapred.TaskController: 547 2015-04-07 03:17:55,933 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201504031414_0032m-1006288762 exited with exit code 143. Number of tasks it ran: 0 548 2015-04-07 03:17:55,929 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201504031414_0032_m_000005_1: 549 java.io.FileNotFoundException: File does not exist: hdfs://192.168.111.240:9000/tmp/hadoop-fedora/mapred/system/job_201504031414_0032/jobToken 550 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:558) 551 at org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4489) 552 at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1285) 553 at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 554 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 555 at java.lang.Thread.run(Thread.java:745) 556 557 2015-04-07 03:17:55,934 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to set finish time for task attempt_201504031414_0032_m_000005_1 when no start time is set, stac kTrace is : java.lang.Exception 558 at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145) 559 at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3326) 560 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2613) 561 at java.lang.Thread.run(Thread.java:745) 562 563 2015-04-07 03:17:55,934 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1 564 2015-04-07 03:17:55,943 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201504031414_0032_m_708372940 565 2015-04-07 03:17:55,946 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201504031414_0032_m_708372940 spawned. 566 2015-04-07 03:17:55,955 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-fedora/mapred/local/ttprivate/taskTracker/fedora/jobcache/job_201504 031414_0032/attempt_201504031414_0032_m_000004_1/taskjvm.sh 567 2015-04-07 03:17:57,345 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 1 568 2015-04-07 03:17:57,345 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 569 2015-04-07 03:17:57,345 INFO org.apache.hadoop.mapred.TaskController: 570 2015-04-07 03:17:57,345 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201504031414_0032_m_708372940 but just removed 571 2015-04-07 03:17:57,345 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201504031414_0032_m_708372940 exited with exit code 1. Number of tasks it ran: 0 572 2015-04-07 03:17:57,346 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201504031414_0032_m_000004_1 : Child Error 573 java.io.IOException: Task process exit with nonzero status of 1. 574 at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) 575 2015-04-07 03:18:00,348 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2 576 2015-04-07 03:22:04,053 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0033_m_000049_1 task's state:UNASSIGNED 577 2015-04-07 03:22:04,053 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0033_m_000049_1 which needs 1 slots 578 2015-04-07 03:22:04,053 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201504031414_0033_m_000049_1 which needs 1 slots 579 2015-04-07 03:22:04,141 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user fedora on this TT. 580 2015-04-07 03:22:06,774 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201504031414_0033_r_000025_0 task's state:UNASSIGNED 581 2015-04-07 03:22:06,775 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201504031414_0033_r_000025_0 which needs 1 slots 582 2015-04-07 03:22:06,775 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 4 and trying to launch attempt_201504031414_0033_r_000025_0 which needs 1 slots 583 2015-04-07 03:22:21,913 INFO org.apache.hadoop.mapred.TaskTracker: Cleanup for id job_201504031414_0033 skipped as its localizing. 584 2015-04-07 03:22:25,383 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201504031414_0033_r_284599715 585 2015-04-07 03:22:25,383 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201504031414_0033_r_284599715 spawned. 586 2015-04-07 03:22:25,384 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201504031414_0033m-1040099847 587 2015-04-07 03:22:25,387 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201504031414_0033m-1040099847 spawned. 588 2015-04-07 03:22:25,390 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-fedora/mapred/local/ttprivate/taskTracker/fedora/jobcache/job_201504 031414_0033/attempt_201504031414_0033_r_000025_0/taskjvm.sh 589 2015-04-07 03:22:25,392 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-fedora/mapred/local/ttprivate/taskTracker/fedora/jobcache/job_201504 031414_0033/attempt_201504031414_0033_m_000049_1/taskjvm.sh 590 2015-04-07 03:22:25,914 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201504031414_0033 591 2015-04-07 03:22:25,914 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201504031414_0033m-1040099847 but just removed 592 2015-04-07 03:22:25,914 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2 593 2015-04-07 03:22:25,915 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201504031414_0033_m_000049_1 not found in cache 594 2015-04-07 03:22:25,915 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201504031414_0033_r_284599715 but just removed 595 2015-04-07 03:22:25,915 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 4 596 2015-04-07 03:22:25,929 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201504031414_0033 for user-log deletion with retainTimeStamp:1428463345916 597 2015-04-07 03:22:27,391 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 1 598 2015-04-07 03:22:27,391 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 599 2015-04-07 03:22:27,391 INFO org.apache.hadoop.mapred.TaskController: 600 2015-04-07 03:22:27,391 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201504031414_0033_r_284599715 exited with exit code 1. Number of tasks it ran: 0 601 2015-04-07 03:22:27,405 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 1 602 2015-04-07 03:22:27,405 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 603 2015-04-07 03:22:27,405 INFO org.apache.hadoop.mapred.TaskController: 604 2015-04-07 03:22:27,405 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201504031414_0033m-1040099847 exited with exit code 1. Number of tasks it ran: 0 605 2015-04-07 03:22:28,917 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201504031414_0033 being deleted. 606 2015-04-07 03:22:28,917 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201504031414_0033 being deleted.

adrian-wang commented 9 years ago

It seems the input path does not exists or cannot be found. Did you run bayes/bin/prepare.sh before you run run.sh or did you get any exception during prepare?

echozyw commented 9 years ago

I did run prepare.sh and it succeeded. When I run run.sh, it successfully completed several jobs and then failed with the following output errors. I have 2 questions here: 1) What caused the "15/04/07 03:22:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_0&filter=stderr 15/04/07 03:22:07 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_1, Status : FAILED Error launching task 15/04/07 03:22:07 WARN mapred.JobClient: Error reading task "?

2) It seems that the run.sh failed at "Exception in thread "main" java.lang.IllegalStateException: Job failed! at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateCollocations(CollocDriver.java:239) at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateAllGrams(CollocDriver.java:188) at ". What should I check for the debugging?

Thanks a lot!

======================here is the error output============================== $./run.sh ...... (successful jobs) 15/04/07 03:16:31 INFO mapred.JobClient: map 95% reduce 0% 15/04/07 03:17:14 INFO mapred.JobClient: map 97% reduce 0% 15/04/07 03:17:30 INFO mapred.JobClient: map 99% reduce 0% 15/04/07 03:17:53 INFO mapred.JobClient: map 100% reduce 0% 15/04/07 03:17:55 INFO mapred.JobClient: Job complete: job_201504031414_0032 15/04/07 03:17:55 INFO mapred.JobClient: Counters: 20 15/04/07 03:17:55 INFO mapred.JobClient: Job Counters 15/04/07 03:17:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1467609 15/04/07 03:17:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/07 03:17:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/07 03:17:55 INFO mapred.JobClient: Rack-local map tasks=13 15/04/07 03:17:55 INFO mapred.JobClient: Launched map tasks=60 15/04/07 03:17:55 INFO mapred.JobClient: Data-local map tasks=47 15/04/07 03:17:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=5694 15/04/07 03:17:55 INFO mapred.JobClient: File Output Format Counters 15/04/07 03:17:55 INFO mapred.JobClient: Bytes Written=87165324 15/04/07 03:17:55 INFO mapred.JobClient: FileSystemCounters 15/04/07 03:17:55 INFO mapred.JobClient: HDFS_BYTES_READ=181494754 15/04/07 03:17:55 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2710790 15/04/07 03:17:55 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=87165324 15/04/07 03:17:55 INFO mapred.JobClient: File Input Format Counters 15/04/07 03:17:55 INFO mapred.JobClient: Bytes Read=181488418 15/04/07 03:17:55 INFO mapred.JobClient: Map-Reduce Framework 15/04/07 03:17:55 INFO mapred.JobClient: Map input records=40000 15/04/07 03:17:55 INFO mapred.JobClient: Physical memory (bytes) snapshot=3825807360 15/04/07 03:17:55 INFO mapred.JobClient: Spilled Records=0 15/04/07 03:17:55 INFO mapred.JobClient: CPU time spent (ms)=117680 15/04/07 03:17:55 INFO mapred.JobClient: Total committed heap usage (bytes)=1525678080 15/04/07 03:17:55 INFO mapred.JobClient: Virtual memory (bytes) snapshot=40598466560 15/04/07 03:17:55 INFO mapred.JobClient: Map output records=40000 15/04/07 03:17:55 INFO mapred.JobClient: SPLIT_RAW_BYTES=6336

15/04/07 03:21:59 INFO input.FileInputFormat: Total input paths to process : 48 15/04/07 03:22:00 INFO mapred.JobClient: Running job: job_201504031414_0033 15/04/07 03:22:01 INFO mapred.JobClient: map 0% reduce 0% 15/04/07 03:22:04 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_0, Status : FAILED Error launching task 15/04/07 03:22:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_0&filter=stdout 15/04/07 03:22:04 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave2:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_0&filter=stderr 15/04/07 03:22:07 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_1, Status : FAILED Error launching task 15/04/07 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_1&filter=stdout 15/04/07 03:22:07 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_1&filter=stderr 15/04/07 03:22:09 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_r_000025_0, Status : FAILED Error launching task 15/04/07 03:22:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_r_000025_0&filter=stdout 15/04/07 03:22:09 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave3:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_r_000025_0&filter=stderr 15/04/07 03:22:12 INFO mapred.JobClient: Task Id : attempt_201504031414_0033_m_000049_2, Status : FAILED Error launching task 15/04/07 03:22:12 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_2&filter=stdout 15/04/07 03:22:12 WARN mapred.JobClient: Error reading task outputhttp://newjobtestSlave1:50060/tasklog?plaintext=true&attemptid=attempt_201504031414_0033_m_000049_2&filter=stderr 15/04/07 03:22:21 INFO mapred.JobClient: Job complete: job_201504031414_0033 15/04/07 03:22:21 INFO mapred.JobClient: Counters: 4 15/04/07 03:22:21 INFO mapred.JobClient: Job Counters 15/04/07 03:22:21 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=5062 15/04/07 03:22:21 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 15/04/07 03:22:21 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 15/04/07 03:22:21 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 Exception in thread "main" java.lang.IllegalStateException: Job failed! at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateCollocations(CollocDriver.java:239) at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateAllGrams(CollocDriver.java:188) at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:183) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:271) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:55) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Warning: $HADOOP_HOME is deprecated.

adrian-wang commented 9 years ago

You HiBench complains Input path does not exist: /home/fedora/Bayes/Output-comp/vectors/tfidf-vectors, which should be hdfs://HiBench/Bayes/Output-comp/vectors/tfidf-vectors This is the problem. Please try to set HADOOP_HOME in your bash environment explicitly, it may be required by earlier mahout versions.

adrian-wang commented 9 years ago

Did your problem get resolved?

echozyw commented 9 years ago

Thank you for checking. But it still has the same issue. I am pretty sure that I already set HADOOP_HOME in my bash environment explicitly, when I ran into those errors. I am giving it another try to see how it goes. Will report back once it is done/dead...

echozyw commented 9 years ago

It is dead as before. The issue seems to be that: The 3rd job (the first 2 jobs seem to have passed) that came along when I do ./run.sh complains that "ERROR security.UserGroupInformation: PriviledgedActionException as:fedora cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: /HiBench/Bayes/Output-comp/vectors/tfidf-vectors". And here is the output when I ls the hadoop folder: [fedora@newjobtestmaster ~]$ hadoop dfs -ls /HiBench/Bayes/Output-comp/vectors Found 2 items drwxr-xr-x - fedora supergroup 0 2015-05-05 04:26 /HiBench/Bayes/Output-comp/vectors/tokenized-documents drwxr-xr-x - fedora supergroup 0 2015-05-05 05:54 /HiBench/Bayes/Output-comp/vectors/wordcount

When/where is the missing file "tfidf-vectors" supposed to be generated? (Is there any way that I can attach my error trace file here? It seems a bit too long to store as an image.)

Thank you very much for your help in advance, Z