HIPI(2.1.0) MapReduce(Hadoop2.7.1) example code issues.

[yangboz]

There are sort of code issues(http://hipi.cs.virginia.edu/gettingstarted.html) on Hadoop Job and JobConf different configurations on input/output/fs properties, workaround is:

    // Initialize and configure MapReduce job
    Configuration configuration = new Configuration(true);
    Job job = Job.getInstance(configuration, "HipiConfig");
    JobConf jobConf = new JobConf(configuration, Job.class);
    job.getConfiguration().set("mapreduce.input.fileinputformat.inputdir", strings[0]);
    job.getConfiguration().set("mapreduce.output.fileoutputformat.outputdir", strings[1]);

And

  job.getConfiguration().set("fs.hdfs.impl",
            org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
    );
    job.getConfiguration().set("fs.file.impl",
            org.apache.hadoop.fs.LocalFileSystem.class.getName()
    );

Then missing class issue:

Caused by: java.lang.NoClassDefFoundError: org/json/simple/JSONValue
java.lang.Exception: java.lang.NoClassDefFoundError: com/drew/imaging/ImageProcessingException

Hopefully find more and thanks for this great works!

[yangboz]

And the workaround for missing class issue is adding missing dependencies:

<dependency>
            <groupId>com.googlecode.json-simple</groupId>
            <artifactId>json-simple</artifactId>
            <version>1.1.1</version>
</dependency>       
<dependency>
            <groupId>com.drewnoakes</groupId>
            <artifactId>metadata-extractor</artifactId>
            <version>2.8.1</version>
 </dependency>

Now the MapReduce code example work like a charm. And my example code please check: https://github.com/yangboz/verdant-octo-woof/blob/master/src/main/java/info/smartkit/eip/hadoop/HipiExample.java

[CarYanG]

I got another class missing :java.lang.ClassNotFoundException: com.adobe.xmp.XMPException

[yangboz]

@CarYanG please add this pom dependency.

<dependency>
    <groupId>com.adobe.xmp</groupId>
    <artifactId>xmpcore</artifactId>
    <version>5.1.2</version>
</dependency>

[yangboz]

Another hadoop2.6(CDH5.8.2) run-time issue: Input FS: local FS Output HIB: project/input.hib Overwrite HIB if it exists: true Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/JSONValue at org.hipi.image.HipiImageHeader.getMetaDataAsBytes(HipiImageHeader.java:296) at org.hipi.image.HipiImageHeader.write(HipiImageHeader.java:393) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:566) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:637) at org.hipi.tools.HibImport.main(HibImport.java:140) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.json.simple.JSONValue at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 11 more

Workaround: sudo cp ~/Downloads/json-simple-1.1.1.jar /usr/lib/hadoop/

[ashishkumar4029]

@yangboz where do we need to add these dependencies?

[yangboz]

@ashishfirst96 it is a runtime lib dependency, JOSNSimple can be added in your pom.xml also.

[Majdouline-Meddad]

can you please tell me the name of the file

[Majdouline-Meddad]

i don't know where i should put this dependency in hipi-release

[Majdouline-Meddad]

majdouline@hadoop:/usr/local/hipi-release/tools$ ./hibImport.sh res res5.hib Input image directory: res Input FS: local FS Output HIB: res5.hib Overwrite HIB if it exists: false Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/JSONValue at org.hipi.image.HipiImageHeader.getMetaDataAsBytes(HipiImageHeader.java:296) at org.hipi.image.HipiImageHeader.write(HipiImageHeader.java:393) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:566) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:637) at org.hipi.tools.HibImport.main(HibImport.java:136) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.json.simple.JSONValue at java.net.URLClassLoader$1.run(URLClassLoader.java:359) at java.net.URLClassLoader$1.run(URLClassLoader.java:348) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:347) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 11 more majdouline@hadoop:/usr/local/hipi-release/tools$ ./hibImport.sh res res6.hib Input image directory: res Input FS: local FS Output HIB: res6.hib Overwrite HIB if it exists: false Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/JSONValue at org.hipi.image.HipiImageHeader.getMetaDataAsBytes(HipiImageHeader.java:296) at org.hipi.image.HipiImageHeader.write(HipiImageHeader.java:393) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:566) at org.hipi.imagebundle.HipiImageBundle.addImage(HipiImageBundle.java:637) at org.hipi.tools.HibImport.main(HibImport.java:136) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.json.simple.JSONValue at java.net.URLClassLoader$1.run(URLClassLoader.java:359) at java.net.URLClassLoader$1.run(URLClassLoader.java:348) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:347) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 11 more

[Majdouline-Meddad]

i don't have pom.xml i'am working by gradle

[ashishkumar4029]

@majdou41 The same problem happened with me. This Exception has occurred because there no json package available in the hipi directory. This issue can simply be resolved by adding the json-simple package into the hipi directory, and then gradle the hipi package. I have attached the json-simple package.

json.zip

I have attached the json package, copy this into the /hipi/tools/hibImport/src/main/java/org directory

you may use this command sudo cp -r <you json package source> </hipi/tools/hibImport/src/main/java/org destinaton>

hope it works, if not then tell me!

[midoBi]

Thanks @ashishfirst96 it work fine Now.

[ashishkumar4029]

@midoBi Happy to help!

[midoBi]

@ashishkumar4029 hi i have other exception when using hipi tool hibDump java class not found when run this commande ./hibDump foderImagesHDFS pathFile java.lang.NoClassDefFoundError: com/drew/imaging/ImageProcessingException

[ashishkumar4029]

@midoBi This exception has occurred because there is no drew package available. I have attached the required package.

com.zip

copy this into the /hipi/tools/hibDump/src/main/java directory

you may use this command sudo cp -r <you json package source> </hipi/tools/hibDump/src/main/java destinaton>

hope it works, if not then tell me!

[abiodungit]

I got a class error, have no clue how to resolve it: hadoop jar ImageCountProjJob2.jar FaceCount TstImage.hib outputX Exception in thread "main" java.lang.NoClassDefFoundError: hipi/imagebundle/mapreduce/ImageBundleInputFormat at FaceCount.run(FaceCount.java:155) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at FaceCount.main(FaceCount.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.lang.ClassNotFoundException: hipi.imagebundle.mapreduce.ImageBundleInputFormat

[Akki357358]

Input image directory: /home/hduser/sample Input FS: local FS Output HIB: outputimages.hib Overwrite HIB if it exists: false Exception in thread "main" java.lang.NullPointerException at java.util.Arrays.sort(Arrays.java:1246) at org.hipi.tools.HibImport.main(HibImport.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

i am getting this error any solution please?

[hajirambee]

I got this error. Dont know how to resolve it? tools/hibImport.sh ~/SampleImages sampleimages.hib tools/runTool.sh: line 10: hadoop: command not found

I am new to Hadoop. Any solution please? Thanks in advance..