search

edwardcapriolo / filecrush

Remedy small files by combining them into larger ones.
193 stars 120 forks source link

/tmp/crush-ee981b79-cb49-4a0b-9d61-f876c349dfd5/out does not exist #11

Open alrustamov opened 8 years ago

alrustamov commented 8 years ago

~/git/filecrush[1131] $ hadoop jar target/filecrush-2.2.2-SNAPSHOT.jar com.m6d.filecrush.crush.Crush /user/arustamov/4 /user/arustamov/crush 20160701073112 16/07/01 07:31:50 INFO Configuration.deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 16/07/01 07:31:50 INFO Configuration.deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 16/07/01 07:31:50 INFO Configuration.deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 16/07/01 07:31:52 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 16/07/01 07:31:52 INFO compress.CodecPool: Got brand-new compressor [.deflate] 16/07/01 07:31:52 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces Exception in thread "main" java.io.FileNotFoundException: File hdfs://nameservice1/user/arustamov/tmp/crush-ee981b79-cb49-4a0b-9d61-f876c349dfd5/out does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:705) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:106) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:763) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:759) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:759) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1497) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537) at com.m6d.filecrush.crush.Crush.getOutputMappings(Crush.java:795) at com.m6d.filecrush.crush.Crush.moveOutput(Crush.java:814) at com.m6d.filecrush.crush.Crush.run(Crush.java:668) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.m6d.filecrush.crush.Crush.main(Crush.java:1330) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

alrustamov commented 8 years ago

$ java -version java version "1.6.0_30" OpenJDK Runtime Environment (IcedTea6 1.13.3) (rhel-5.1.13.3.el6_5-x86_64) OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)`

$ hadoop version Hadoop 2.6.0-cdh5.7.0 Subversion http://github.com/cloudera/hadoop -r c00978c67b0d3fe9f3b896b5030741bd40bf541a Compiled by jenkins on 2016-03-23T18:36Z Compiled with protoc 2.5.0 From source with checksum b2eabfa328e763c88cb14168f9b372 This command was run using /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/hadoop-common-2.6.0-cdh5.7.0.jar

devorezr commented 7 years ago

Did this ever get solved? I am having the same problem.

edwardcapriolo commented 7 years ago

It does not seem like it. If there is a patch out there I can give it a try. (I confess to being out of touch with this for a while)