search

deva-rajan / hamake

Automatically exported from code.google.com/p/hamake
0 stars 0 forks source link

Hamake creates redundant amount of Hadoop Configuration objects #31

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
10/05/23 18:06:45 ERROR task.MapReduce: Failed to execute Hadoop command 
file:/home/project/Fb2Pdf/src/fb2pdf/src/java/dist/fb2pdf.jar/com.fb2pdf.ha
doop.MeanShiftCanopyDriver
java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop-
0.20/conf.empty/core-site.xml (Too many open files)
    at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1352)
    at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1217)
    at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1156)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:450)
    at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:248)
    at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:266)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:179)
    at 
com.fb2pdf.hadoop.MeanShiftCanopyDriver.createCanopyFromVectors(MeanShiftCa
nopyDriver.java:112)
    at 
com.fb2pdf.hadoop.MeanShiftCanopyDriver.main(MeanShiftCanopyDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
9)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    at com.codeminders.hamake.task.MapReduce.execute(MapReduce.java:85)
    at com.codeminders.hamake.dtr.Fold.execute(Fold.java:90)
    at 
com.codeminders.hamake.dtr.DataTransformationRule.executeIfCan(DataTransfor
mationRule.java:75)
    at com.codeminders.hamake.TaskThread.run(TaskThread.java:47)

Original issue reported on code.google.com by vorl.s...@gmail.com on 23 May 2010 at 4:11

GoogleCodeExporter commented 8 years ago 
The problem is in "main" method of Hadoop "RunJar" class - it doesn't frees 
file 
descriptors that it opens during loading of libraries

Original comment by vorl.s...@gmail.com on 25 May 2010 at 10:45

GoogleCodeExporter commented 8 years ago 
http://blogs.sun.com/CoreJavaTechTips/entry/closing_a_urlclassloader

Original comment by vorl.s...@gmail.com on 25 May 2010 at 9:20

GoogleCodeExporter commented 8 years ago 
fixed by r300

Original comment by abon...@gmail.com on 1 Jun 2010 at 12:58

GoogleCodeExporter commented 8 years ago 
It seems that ClassLoader also loads and does not free jar file by itself. 
There still exist problem with file descriptors leak

Original comment by vorl.s...@gmail.com on 8 Jun 2010 at 2:11

GoogleCodeExporter commented 8 years ago 
fixed in r317

Original comment by vorl.s...@gmail.com on 9 Jun 2010 at 4:15