search

forcedotcom / phoenix

BSD 3-Clause "New" or "Revised" License
558 stars 227 forks source link

Non IO Exception in readFields doesn't prevent from retrying when it should. #628

Open samarthjain opened 10 years ago

samarthjain commented 10 years ago

HBase keeps retrying even when a non-IO exception like InstatiationException is thrown in SingleKeyValueComparisonFilter.readFields method. 


java.lang.RuntimeException: java.lang.InstantiationException: com.salesforce.phoenix.expression.function.FloorDateExpression
    at com.salesforce.phoenix.expression.ExpressionType.newInstance(ExpressionType.java:195)
    at com.salesforce.phoenix.expression.BaseCompoundExpression.readFields(BaseCompoundExpression.java:92)
    at com.salesforce.phoenix.expression.ComparisonExpression.readFields(ComparisonExpression.java:136)
    at com.salesforce.phoenix.filter.BooleanExpressionFilter.readFields(BooleanExpressionFilter.java:112)
    at com.salesforce.phoenix.filter.SingleKeyValueComparisonFilter.readFields(SingleKeyValueComparisonFilter.java:145)
    at org.apache.hadoop.hbase.client.Scan.readFields(Scan.java:592)
    at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:690)
    at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1311)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.InstantiationException: com.salesforce.phoenix.expression.function.FloorDateExpression
    at java.lang.Class.newInstance0(Class.java:340)
    at java.lang.Class.newInstance(Class.java:308)
    at com.salesforce.phoenix.expression.ExpressionType.newInstance(ExpressionType.java:193)
    ... 15 more
samarthjain commented 10 years ago

Turns out even by wrapping the Throwable and re-throwing it as DoNotRetryIOException doesn't help. This is because of a bug in HbaseObjectWritable.readObject method where it catches exceptions and re-throws them back as regular IO Exceptions. Filed a jira issue: https://issues.apache.org/jira/browse/HBASE-10181

jtaylor-sfdc commented 10 years ago

@samarthjain - Lars has fixed the underlying HBase issue (I believe in HBase 0.94.15). Would you mind sending me a pull for the Phoenix part of this (i.e. the wrapping in a DoNotRetryIOException) and verifying that the problem is fixed in 0.94.15?

jtaylor-sfdc commented 10 years ago

@samarthjain ?