search

fullcontact / hadoop-sstable

Splittable Input Format for Reading Cassandra SSTables Directly
Apache License 2.0
49 stars 14 forks source link

SequenceFile doesn't work with GzipCodec without native-hadoop code! #11

Closed gadodia closed 9 years ago

gadodia commented 9 years ago

java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code! at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476) at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:569) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)

Is anyone familiar with this exception? I am running it on mac os.

bvanberg commented 9 years ago

You'll need to install the native libraries hadoop needs to run with gzip. Have a look here: https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/MaHLwF1mtvI

bvanberg commented 9 years ago

What hadoop distro are you running?

bvanberg commented 9 years ago

This is an issue with running hadoop on mac os x with compression enabled. You could either run without compression, or do something like this: http://gauravkohli.com/2014/09/28/building-native-hadoop-v-2-4-1-libraries-for-os-x/

gadodia commented 9 years ago

Thanks..... installed native library and pointed eclipse to that library and it worked.