search

aseldawy / spatialhadoop2

The second generation of SpatialHadoop that ships as an extension
Other
153 stars 98 forks source link

HDP Issue with Large File #19

Open Harvail opened 9 years ago

Harvail commented 9 years ago

I am getting following error, this error only occurs when I run with larger file e.g. shadoop generate test.rects size:1.gb shape:rect mbr:0,0,1000,1000

if I run with small file e.g shadoop generate test.rects size:1.mb shape:rect mbr:0,0,1000,1000 This works fine. I am using HDP-2.3.0.0-2557 (HortonWorks) sandbox runs on CentOS

I already copied files to /usr/hdp/2.3.0.0-2557/hadoop/lib /usr/hdp/2.3.0.0-2557/hadoop/bin /usr/hdp/2.3.0.0-2557/hadoop/client /usr/hdp/2.3.0.0-2557/hadoop/conf

Error: java.lang.ClassNotFoundException: com.vividsolutions.jts.io.ParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at edu.umn.cs.spatialHadoop.core.Rectangle.fromText(Rectangle.java:274)
at edu.umn.cs.spatialHadoop.OperationsParams.getTextSerializable(OperationsParams.java:418)
at edu.umn.cs.spatialHadoop.OperationsParams.getShape(OperationsParams.java:357)
at edu.umn.cs.spatialHadoop.OperationsParams.getShape(OperationsParams.java:352)
at edu.umn.cs.spatialHadoop.mapred.RandomShapeGenerator.(RandomShapeGenerator.java:77)
at edu.umn.cs.spatialHadoop.mapred.RandomInputFormat.getRecordReader(RandomInputFormat.java:90)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

aseldawy commented 9 years ago

This error indicates that JTS library is not loaded by Hadoop. It happens only with large files because small files are processed using a single-machine algorithm, for efficiency, and do not go through Hadoop MapReduce at all. Make sure that the jts-1.8.jar file is in the '/lib' directory (for Hadoop 1.x) or in the '/share/hadoop/common/lib/' (for Hadoop 2.x). Keep in mind that you also need to restart the cluster once for the libraries to be loaded by Hadoop.