MapReduce & Hadoop

[HorizonNet]

Below is a short review.

Tasks

• [x] Download a book text from the Gutenberg Project and copy it into HDFS
• [x] Write a MapReduce program that does a WordCount on an arbitrary text file
• [x] Package your WordCount application into a Maven project and build it
• [x] Using teragen and terasort

Below are a few comments/suggestions:

• The Hadoop version (0.20.2) in your Maven project is very old. You should use a recent version (at least 2.6.x, better 2.7.x). It also doesn't match the version (2.8.0) from your HDFS tests.
• When writing Hadoop code the scope of Hadoop dependencies should be set to provided, because it is already available on the platform and prevents conflicts.
• Try to use the Maven standards for versioning (f.e. 1.0.0-SNAPSHOT).
• Add a .gitignore file to your project. You don't want to have certain files in your repository, such as the target directory.
• In further exercises you should add the complete output of a command. This one is missing in the HDFS tests.

Summary:

You're done with this one. Good work.

[fabianrbz]

@HorizonNet thanks for the feedback! I didn't realized about hadoop's version in the maven project. Honestly I ran everything from the command line using version 2.8.0 and then built the maven project, but apparently I put the wrong version in the pom.xml

Do you recommend/use any specific IDE to work with maven? It's been a while since I use java, so I'm not sure which tool is best. Also I kind of prefer the command line, but I know that an IDE makes your life easier when working with java

[HorizonNet]

I also love the command line and do most of the stuff on it. Regarding the coding stuff via an IDE, I prefer IntelliJ IDEA, but some also use Eclipse. The choice depends on your personal preferences.