search

mozilla-metrics / akela

A bunch of utility classes for Java, Hadoop, HBase, Pig, etc.
Apache License 2.0
76 stars 31 forks source link

Many java.io.IOException: Filesystem closed #5

Open ylangisc opened 12 years ago

ylangisc commented 12 years ago

Apologies for directly opening an issue but I didn't found a way to just drop a question.

I've tried to backup parts of my hdfs with your class and I constantly get several IOException (see below). Just using distcp works fine. Any ideas?

Thanks Yves

[ec2-user@ip-10-40-211-16 ~]$ sudo /usr/local/hadoop-1.0.0/bin/hadoop jar akela-0.5-SNAPSHOT-job.jar com.mozilla.hadoop.Backup hdfs://ip-10-40-211-16.ec2.internal:8020/hbase/logfile local2 /usr/local/hbase-0.92.0/hbase-0.92.0.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.3.1.jar:/usr/local/hbase-0.92.0/conf:/usr/local/hbase-0.92.0/lib/guava-r09.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.4.2.jar Adding input path: Backup-inputsource0.txt Adding input path: Backup-inputsource1.txt 12/11/11 10:06:16 INFO input.FileInputFormat: Total input paths to process : 2 12/11/11 10:06:16 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 12/11/11 10:06:16 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev c7d54fffe5a853c437ee23413ba71fc6af23c91d] 12/11/11 10:06:16 INFO mapred.JobClient: Running job: job_201210221953_0008 12/11/11 10:06:17 INFO mapred.JobClient: map 0% reduce 0% 12/11/11 10:06:31 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_0, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:34 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_0, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:37 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_1, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:40 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_1, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:43 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_2, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:46 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_2, Status : FAILED java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:55 INFO mapred.JobClient: Job complete: job_201210221953_0008 12/11/11 10:06:55 INFO mapred.JobClient: Counters: 7 12/11/11 10:06:55 INFO mapred.JobClient: Job Counters 12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=50404 12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 12/11/11 10:06:55 INFO mapred.JobClient: Launched map tasks=8 12/11/11 10:06:55 INFO mapred.JobClient: Data-local map tasks=8 12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 12/11/11 10:06:55 INFO mapred.JobClient: Failed map tasks=1`

xstevens commented 12 years ago

From what it looks like. I think the problem is that you're running this on the same cluster for input/output. There's some assumptions (perhaps bad ones) in Backup that you're going from one cluster to another. If you wanted to fix this for your case take a look at the setup and closing of inputFs and outputFs. The other thing to watch out for is "fs.automatic.close" could be messing with you on inputFs, but I'm not sure without trying to debug it.

Just FYI, HBase has some good tools for this type of thing now as well. Take a look at their Export/Import jobs.