Running into com.fullcontact.cassandra.io.sstable.CorruptBlockException

[ggjoshi]

I am trying to decode some sstable files with following two steps:-

Indexing step:- hadoop jar hadoop-sstable-0.1.4.jar com.fullcontact.sstable.index.SSTableIndexIndexer /data//cassandra-data

Decoding step, which needs the create table statement: hadoop jar hadoop-sstable-0.1.4.jar com.fullcontact.sstable.example.SimpleExample -D hadoop.sstable.cql="CREATE TABLE analytics_counters_daily (access_attribute varchar,matrix_type varchar,time_bucket timestamp,event_type varchar,platform varchar,position varchar,attribute_key varchar,medium varchar,count counter,PRIMARY KEY ( (access_attribute, matrix_type), time_bucket, event_type, platform, position, attribute_key, medium)) WITH CLUSTERING ORDER BY (time_bucket DESC);" /data//cassandra-data /data//cassandra-data/decoded

But I get a bunch of corrupt sstable exceptions and the MR job to decode cassandra data fails. Any suggestion on what may be wrong?

15/10/28 05:25:23 INFO mapred.JobClient: Task Id : attempt_201506251917_1703_m_000080_1, Status : FAILED org.apache.cassandra.io.sstable.CorruptSSTableException: com.fullcontact.cassandra.io.sstable.CorruptBlockException: (hdfs://watson-batch-hbase-namenode-prod/data/gjoshi/cassandra-data/analytics_counters_daily-1f77832036fd11e5a007a7e74a491f29/analytics_counters_backfill-analytics_counters_daily-ka-1305-Data.db): corruption detected, chunk at 1450382757 of length 7744. at com.fullcontact.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:86) at com.fullcontact.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:421) at com.fullcontact.sstable.hadoop.mapreduce.SSTableRecordReader.initialize(SSTableRecordReader.java:67) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:479) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) a

[eentzel]

@ggjoshi what version of Cassandra are you using? The master branch is for C* 1.x — if you're using 2.x there's a cassandra-2.0.x brach with alpha support.

[ggjoshi]

@eentzel thanks for your response. I am using cassandra 2.1.6. Per your suggestion I built the jar from cassandra-2.0.x branch and tried running the following commands on files I copied from my cassandra cluster:-

hadoop jar hadoop-sstable-2.0.0.jar com.fullcontact.sstable.index.SSTableIndexIndexer /data/gjoshi/cassandra-data/analytics_counters_daily-1f77832036fd11e5a007a7e74a491f29

hadoop jar hadoop-sstable-2.0.0.jar com.fullcontact.sstable.example.SimpleExample -D hadoop.sstable.cql="CREATE TABLE analytics_counters_daily (access_attribute varchar,matrix_type varchar,time_bucket timestamp,event_type varchar,platform varchar,position varchar,attribute_key varchar,medium varchar,count counter,PRIMARY KEY ( (access_attribute, matrix_type), time_bucket, event_type, platform, position, attribute_key, medium)) WITH CLUSTERING ORDER BY (time_bucket DESC);" -D mapred.task.timeout=21600000 -D mapred.map.tasks.speculative.execution=false -D mapred.job.reuse.jvm.num.tasks=1 -D io.sort.mb=1000 -D io.sort.factor=100 -D mapred.reduce.tasks=512 -D hadoop.sstable.split.mb=1024 -D mapred.child.java.opts="-Xmx2G -XX:MaxPermSize=256m" /data/gjoshi/cassandra-data/analytics_counters_daily-1f77832036fd11e5a007a7e74a491f29 /data/gjoshi/cassandra-data/decoded

On one of the clusters I get exceptions saying "Can not seek after EOF"

15/10/28 18:24:10 INFO mapred.JobClient: Task Id : attempt_201506251917_1714_m_000070_1, Status : FAILED java.io.IOException: Cannot seek after EOF at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1133) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:48) at com.fullcontact.sstable.hadoop.mapreduce.SSTableSplit.initialize(SSTableSplit.java:82) at com.fullcontact.sstable.hadoop.mapreduce.SSTableRecordReader.initialize(SSTableRecordReader.java:62) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:479) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262)

On one other hadoop cluster, I get the following error

java.lang.IllegalArgumentException: Expected data file in jb format, but got: hdfs://watson-realtime-hbase-namenode-prod/data/gjoshi/cassandra-data/analytics_counters_daily-1f77832036fd11e5a007a7e74a491f29/analytics_counters_backfill-analytics_counters_daily-ka-988-Data.db at com.fullcontact.sstable.hadoop.mapreduce.SSTableRecordReader.initialize(SSTableRecordReader.java:67) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:479) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262)

Do you know why this may be happening?

[Xorlev]

@bvanberg is ka 2.1.x?

[bvanberg]

Nope. 1.2, 2.0.x only at the moment. We don't yet use 2.1.x internally so have not upgraded.

[ggjoshi]

@Xorlev @bvanberg does that mean the sstable files from 2.1.x are not supported by hadoop-sstable?

[bvanberg]

@ggjoshi That is correct.

[ggjoshi]

@bvanberg thanks for clarifying. Are there any plans to support this scenario(decoding sstable files from 2.1.x) in near future?

[bvanberg]

@ggjoshi We don't have any current plans as we are not yet on 2.1.x ourselves. We are open to PRs, however. I can take a look at what changes are required for 2.1.x and see if it will be a lot of work or not.

[ggjoshi]

alright @bvanberg