search

RevolutionAnalytics / RHadoop

RHadoop
https://github.com/RevolutionAnalytics/RHadoop/wiki
763 stars 278 forks source link

java.io.EOFException #217

Open ddskyfuyu opened 9 years ago

ddskyfuyu commented 9 years ago

I put some simple commands, but Rhaoop can not excute, but I don't know that the reason for these errors, Please help me, thanks.

train.hdfs<-to.dfs(1:10) 14/11/17 15:27:08 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 14/11/17 15:27:08 INFO compress.CodecPool: Got brand-new compressor [.deflate] from.dfs(train.hdfs) 14/11/17 15:27:55 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 14/11/17 15:27:55 INFO compress.CodecPool: Got brand-new decompressor [.deflate] $key NULL

$val [1] 1 2 3 4 5 6 7 8 9 10

train.mr<-mapreduce( train.hdfs, map = function(k, v) { keyval(1,v) }, reduce=function(k,v){ cbind(k,v) }, output.format="csv", output="/user/bre/fy/result")

报错： 4/11/17 15:29:32 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead. packageJobJar: [/tmp/RtmphmGn1f/rmr-local-env88b447258de, /tmp/RtmphmGn1f/rmr-global-env88b4721ec811, /tmp/RtmphmGn1f/rmr-streaming-map88b415706a60, /tmp/RtmphmGn1f/rmr-streaming-reduce88b4d66b8d9, /tmp/hadoop-bre/hadoop-unjar5460590707984984747/] [] /tmp/streamjob968108023756961359.jar tmpDir=null 14/11/17 15:29:34 INFO client.RMProxy: Connecting to ResourceManager at /192.168.44.14:8032 14/11/17 15:29:35 INFO client.RMProxy: Connecting to ResourceManager at /192.168.44.14:8032 14/11/17 15:29:36 INFO mapred.FileInputFormat: Total input paths to process : 1 14/11/17 15:29:36 INFO mapreduce.JobSubmitter: number of splits:2 14/11/17 15:29:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1416031036349_0067 14/11/17 15:29:36 INFO impl.YarnClientImpl: Submitted application application_1416031036349_0067 14/11/17 15:29:36 INFO mapreduce.Job: The url to track the job: http://bjlg-44p14-hadoop01.bfdabc.com:8088/proxy/application_1416031036349_0067/ 14/11/17 15:29:36 INFO mapreduce.Job: Running job: job_1416031036349_0067 14/11/17 15:29:42 INFO mapreduce.Job: Job job_1416031036349_0067 running in uber mode : false 14/11/17 15:29:42 INFO mapreduce.Job: map 0% reduce 0% 14/11/17 15:29:48 INFO mapreduce.Job: Task Id : attempt_1416031036349_0067_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:334) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.typedbytes.TypedBytesInput.readRawBytes(TypedBytesInput.java:218) at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:152) at org.apache.hadoop.streaming.io.TypedBytesOutputReader.readKeyValue(TypedBytesOutputReader.java:51) at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:376)

14/11/17 15:29:48 INFO mapreduce.Job: Task Id : attempt_1416031036349_0067_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:334) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.typedbytes.TypedBytesInput.readRawBytes(TypedBytesInput.java:218) at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:152) at org.apache.hadoop.streaming.io.TypedBytesOutputReader.readKeyValue(TypedBytesOutputReader.java:51) at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:376)

LOG日志：

Logged in as: dr.who About Apache Hadoop Application

About
Jobs

Tools

Log Type: stderr

Log Length: 1077

Loading objects: .Random.seed small.ins train train.dfs train.hdfs train.mr Loading objects: backend.parameters combine combine.file combine.line debug default.input.format default.output.format in.folder in.memory.combine input.format libs map map.file map.line out.folder output.format pkg.opts postamble preamble profile.nodes reduce reduce.file reduce.line rmr.global.env rmr.local.env save.env vectorized.reduce verbose work.dir Loading required package: rmr2 Loading required package: Rcpp Loading required package: RJSONIO Loading required package: bitops Loading required package: digest Loading required package: reshape2 Loading required package: stringr Loading required package: plyr Loading required package: caTools Loading required package: methods log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Log Type: stdout

Log Length: 0

Log Type: syslog

Log Length: 7122

Showing 4096 bytes of 7122 total. Click here for the full log.

apTask: (EQUATOR) 0 kvi 26214396(104857584) 2014-11-17 15:29:04,821 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2014-11-17 15:29:04,821 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2014-11-17 15:29:04,822 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2014-11-17 15:29:04,822 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2014-11-17 15:29:04,842 INFO [main] org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/Rscript, --vanilla, ./rmr-streaming-map88b415706a60] 2014-11-17 15:29:04,917 INFO [main] org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 2014-11-17 15:29:05,926 WARN [Thread-13] org.apache.hadoop.streaming.PipeMapRed: java.io.EOFException 2014-11-17 15:29:05,926 INFO [Thread-14] org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done 2014-11-17 15:29:05,927 INFO [main] org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed! java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:334) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.typedbytes.TypedBytesInput.readRawBytes(TypedBytesInput.java:218) at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:152) at org.apache.hadoop.streaming.io.TypedBytesOutputReader.readKeyValue(TypedBytesOutputReader.java:51) at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:376) 2014-11-17 15:29:05,929 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:334) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.typedbytes.TypedBytesInput.readRawBytes(TypedBytesInput.java:218) at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:152) at org.apache.hadoop.streaming.io.TypedBytesOutputReader.readKeyValue(TypedBytesOutputReader.java:51) at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:376)

2014-11-17 15:29:05,933 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2014-11-17 15:29:05,938 WARN [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://192.168.44.14:8020/user/bre/fy/result/_temporary/1/_temporary/attempt_1416031036349_0067_m_000001_0