search

facebookarchive / hive-io-experimental

Hive I/O Library
Other
66 stars 30 forks source link

Filesystem crash #12

Closed nitay closed 11 years ago

nitay commented 11 years ago

I repeatedly see a crash as below. Siying suggested that there may be a place in your code where

FileSystem fs = FileSystem.get(conf);

but not

FileSystem fs = FileSystem.get(path, conf);

because he identified that hdfs://dfs2.data.facebook.com:9000 is the default fs.

I do not have access to your code so cannot debug further. Please have a look.

Job

http://hadoop3658.frc1.facebook.com.:50033/tasklog?taskid=attempt_201303211723.7965_0001_m_000099_0&all=true

Greg

2013-04-09 19:16:04,566 INFO org.apache.hadoop.mapred.DirectTaskUmbilical: Creating direct umbilical to hadoop5338.frc1.facebook.com/10.179.22.53:64569 2013-04-09 19:16:04,642 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2013-04-09 19:16:05,074 INFO org.apache.hadoop.fs.FileSystem: Initialized new filesystem pointing to hdfs://dfs2.data.facebook.com:9000 with the actual address dfs2-0.data.facebook.com:9000 2013-04-09 19:16:05,133 INFO org.apache.hadoop.fs.FileSystem: Initialized new filesystem pointing to hdfs://dfs8.data.facebook.com:9000/user/facebook/warehouse/malewicz_entities_active_places_2013_04_01/_temp with the actual address dfs8-0.data.facebook.com:9000 2013-04-09 19:16:05,144 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2013-04-09 19:16:05,148 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b7347df 2013-04-09 19:16:05,236 INFO org.apache.hadoop.mapred.Task: Cleaning up job 2013-04-09 19:16:05,236 INFO org.apache.hadoop.mapred.Task: Committing job 2013-04-09 19:16:05,251 INFO org.apache.hadoop.fs.FileSystem: File /user/facebook/warehouse/malewicz_entities_active_places_2013_04_01/_temp/_temporary is being deleted only through Trash org.apache.hadoop.fs.FsShell.delete because all deletes must go through Trash. 2013-04-09 19:16:05,335 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.IllegalArgumentException: Wrong FS: hdfs://dfs8.data.facebook.com:9000/user/facebook/warehouse/malewicz_entities_active_places_2013_04_01/_temp, expected: hdfs://dfs2.data.facebook.com:9000 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:354) at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:107) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:164) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:678) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:366) at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:845) at com.facebook.giraph.hive.common.FileSystems.move(FileSystems.java:108) at com.facebook.giraph.hive.output.HiveApiOutputCommitter.noPartitionsCopyData(HiveApiOutputCommitter.java:169) at com.facebook.giraph.hive.output.HiveApiOutputCommitter.commitJob(HiveApiOutputCommitter.java:79) at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1007) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:294) at org.apache.hadoop.mapred.Child.main(Child.java:162) 2013-04-09 19:16:05,337 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2013-04-09 19:16:05,339 WARN org.apache.hadoop.mapred.FileOutputCommitter: Deleting output in hdfs://dfs8.data.facebook.com:9000/user/facebook/warehouse/malewicz_entities_active_places_2013_04_01/_temp/_temporary/_attempt_201303211723.7965_0001_m_000099_0 returns false