KITE-1038: Allow to Optimize by applying constraints on the server if the underlying dataset supports it

[prazanna]

HBaseServerSideFilterable SPI allows clients to set Server side filters to a HBase Scan using the EntityScannerBuilder interface

[rdblue]

Hey, I haven't had a long look at this, but I'm not sure about the newReader modification. I think there might be a better API that works with Constraints more. I haven't though it completely through, but I wanted to get you some feedback to start thinking about. I'm at OSCON for the rest of this week and will try to follow up. Thanks, @prazanna!

[prazanna]

Hey @rdblue, Thanks for taking a look. Yes. I will think about this a little more and may be propose a different API using Constraints.

[rdblue]

Thanks @prazanna! I'm really sorry to give you that feedback without a more though-out fix. Something involving constraints would be great, possibly a registration process that can apply a server-side filter when it sees a particular constraint. Then we could have a way to get the residual constraints (those not implemented server-side) that should be applied client-side. That would be great!

[prazanna]

Hey @rdblue, Sorry it took some time before I can circle around to this. So I have given a shot at optimizing constraints to apply at server-side when the underlying dataset reader supports it.

I am not very happy with the current design, any other alternative has some equal positives and negatives. So I thought I will bounce this off you and hear if you have anything to say on this.

Like:

• Applying server side filters is transparent to the client
• Minimal change to the source code
• Server side filters are applied as pure optimization and does not affect correctness

Dont like:

• Datasets may support a wide variety of Predicates on the server side, currently I am restricting to In with single value and Exists predicates only. This is not too convincing for me, but supporting multiple values for In predicate requires significant changes to the Filter list and i cannot think of a generic way to support RangeFilters etc. Can we support this in the next iteration?
• Naming of the classes. Cant come up with better semantic names for the SPI interfaces. ServerSideConstrainableReader
• Dont have a good way to test it without libraries like mockito. I am not sure if bringing in mockito into kite-data-hbase tests just for this test is okay? If you have any suggestions, please let me know.

Would like your thoughts on it when you have time. Thanks.

[rdblue]

I like the direction you're headed with this, it's looking great! From quickly looking at the filter package in HBase, I think we should be able to push most of the supported predicates to the server side, but it's fine to just support a few for optimization right now. And feel free to bring in mockito! I think we use it elsewhere, and I'm all for better tests.