Hadoop-Snappy enables Snappy compression for Hadoop.

http://code.google.com/p/hadoop-snappy/

This project is integrated into Hadoop Common (JUN 2011).

Hadoop-Snappy can be used as an add-on for recent (released) versions of Hadoop that do not provide Snappy Codec support yet.

Hadoop-Snappy is being kept in synch with Hadoop Common.

Build Hadoop Snappy

1. Requirements: gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3

2. Build/install Snappy (http://code.google.com/p/snappy/)

3. Build Hadoop Snappy

   $ mvn package [-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR]

'snappy.prefix' by default is '/usr/local'. If Snappy is installed in other location than user local set 'snappy.prefix' to the right location.

The built tarball is at target/hadoop-snappy-0.0.1-SNAPSHOT.tar.gz. The tarball includes snappy native library

Install Hadoop Snappy in Hadoop

1. Expand hadoop-snappy-0.0.1-SNAPSHOT.tar.gz file

Copy (recursively) the lib directory of the expanded tarball in the /lib of all Hadoop nodes

$ cp -r hadoop-snappy-0.0.1-SNAPSHOT/lib/* /lib

IMPORTANT: Hadoo Snappy 0.0.1-SNAPSHOT tarball includes Snappy native library.

2. Add the following key/value pairs into core-site.xml

   io.compression.codecs org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, org.apache.hadoop.io.compress.SnappyCodec

3. Restart Hadoop.

License

Hadoop Snappy is licensed under the the Apache License, Version 2.0. Snappy is licensed under the the Apache License, Version 2.0.

Origins

This project is based on the Hadoop LZO codec classes.

The authors of the Hadoop LZO codec classes gave permission to relicense as the Apache License 2.0 (Todd Lipcon, Kevin Weil, Owen O'Malley, Hong Tang, Chris Douglas, Arun C Murthy) Thanks!

http://code.google.com/p/hadoop-gpl-compression https://github.com/kevinweil/hadoop-lzo