search

CarnationWang23 / hyracks

Automatically exported from code.google.com/p/hyracks
Apache License 2.0
0 stars 0 forks source link

closeQuietly error. #98

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Sometimes when I run a job I get the following error 

Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.commons.io.IOUtils.closeQuietly(Ljava/io/Closeable;)V

If I rerun the job it completes without any errors. 

TRIAL 1:

$ bin/pregelix examples/pregelix-example-0.2.3-jar-with-dependencies.jar 
edu.uci.ics.pregelix.example.PageRankVertex -inputpaths /user/vishal/webmap 
-outputpath /tmp/pg_result -i
p `bin/getip.sh` -port 3099 -vnum 20 -num-iteration 5
13/04/10 11:19:54 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
13/04/10 11:19:54 INFO driver.Driver: job started
13/04/10 11:19:55 INFO driver.Driver: jar packing finished 1310ms
13/04/10 11:19:56 INFO driver.Driver: jar deployment finished 1064ms
13/04/10 11:19:57 INFO input.FileInputFormat: Total input paths to process : 1
Apr 10, 2013 11:19:57 AM edu.uci.ics.pregelix.core.jobgen.JobGen 
generateLoadingJob
INFO: number of splits: 1
Apr 10, 2013 11:19:57 AM edu.uci.ics.pregelix.core.jobgen.JobGen 
generateLoadingJob
INFO: hdfs://jarvis.ics.uci.edu:8020/user/vishal/webmap/webmap_link.txt:0+325
Apr 10, 2013 11:19:57 AM edu.uci.ics.hyracks.hdfs.scheduler.Scheduler 
getLocationConstraints
INFO: Data local rate: 1.0
13/04/10 11:19:58 INFO driver.Driver: data loading finished 1218ms
13/04/10 11:19:58 INFO driver.Driver: iteration 1 finished 618ms
13/04/10 11:19:59 INFO driver.Driver: iteration 2 finished 511ms
13/04/10 11:19:59 INFO driver.Driver: iteration 3 finished 338ms
13/04/10 11:19:59 INFO driver.Driver: iteration 4 finished 319ms
13/04/10 11:20:01 INFO driver.Driver: iteration 5 finished 1154ms
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.commons.io.IOUtils.closeQuietly(Ljava/io/Closeable;)V
        at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:986)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:471)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:662)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:706)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:535)
        at java.io.DataInputStream.readBoolean(DataInputStream.java:242)
        at edu.uci.ics.pregelix.dataflow.util.IterationUtils.readTerminationState(IterationUtils.java:131)
        at edu.uci.ics.pregelix.core.driver.Driver.runJob(Driver.java:142)
        at edu.uci.ics.pregelix.example.client.Client.run(Client.java:74)
        at edu.uci.ics.pregelix.example.PageRankVertex.main(PageRankVertex.java:219)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

RETRY:

$ bin/pregelix examples/pregelix-example-0.2.3-jar-with-dependencies.jar 
edu.uci.ics.pregelix.example.PageRankVertex -inputpaths /user/vishal/webmap 
-outputpath /tmp/pg_result -ip `bin/getip.sh` -port 3099 -vnum 20 
-num-iteration 5
13/04/10 11:20:07 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
13/04/10 11:20:08 INFO driver.Driver: job started
13/04/10 11:20:09 INFO driver.Driver: jar packing finished 1294ms
13/04/10 11:20:10 INFO driver.Driver: jar deployment finished 1069ms
13/04/10 11:20:10 INFO input.FileInputFormat: Total input paths to process : 1
Apr 10, 2013 11:20:10 AM edu.uci.ics.pregelix.core.jobgen.JobGen 
generateLoadingJob
INFO: number of splits: 1
Apr 10, 2013 11:20:10 AM edu.uci.ics.pregelix.core.jobgen.JobGen 
generateLoadingJob
INFO: hdfs://jarvis.ics.uci.edu:8020/user/vishal/webmap/webmap_link.txt:0+325
Apr 10, 2013 11:20:10 AM edu.uci.ics.hyracks.hdfs.scheduler.Scheduler 
getLocationConstraints
INFO: Data local rate: 1.0
13/04/10 11:20:11 INFO driver.Driver: data loading finished 861ms
13/04/10 11:20:12 INFO driver.Driver: iteration 1 finished 530ms
13/04/10 11:20:12 INFO driver.Driver: iteration 2 finished 444ms
13/04/10 11:20:12 INFO driver.Driver: iteration 3 finished 357ms
13/04/10 11:20:13 INFO driver.Driver: iteration 4 finished 309ms
13/04/10 11:20:13 INFO driver.Driver: iteration 5 finished 264ms
13/04/10 11:20:14 INFO driver.Driver: result writing finished 640ms
13/04/10 11:20:14 INFO driver.Driver: job finished

What version of the product are you using? On what operating system?
Version 0.2.3 with CDH4.2

Vishal

Original issue reported on code.google.com by write2vi...@gmail.com on 10 Apr 2013 at 6:24

GoogleCodeExporter commented 8 years ago 
OK, so this error seems persistent now. I'm not able to run pregelix job with 
more than 90 supersteps (sometimes even less) before it fails with this error. 

This is the case only when reading input from HDFS or writing output to HDFS. 

As mentioned before, this is not consistent, and occurs at the end of some 
random superstep! 

Vishal

Original comment by write2vi...@gmail.com on 9 May 2013 at 5:06

GoogleCodeExporter commented 8 years ago

Original comment by salsuba...@gmail.com on 9 May 2013 at 5:07

GoogleCodeExporter commented 8 years ago 
It looks to be a version issue of org.apache.commons.io.IOUtils.
Let me check.

Original comment by buyingyi@gmail.com on 9 May 2013 at 8:44