Introduce more rules on predicate resolution

sadikovi / spark-netflow

NetFlow data source for Spark SQL and DataFrames

Apache License 2.0

18 stars 11 forks source link

Introduce more rules on predicate resolution #52

Open sadikovi opened 7 years ago

sadikovi commented 7 years ago

Currently we support predicate resolution based on statistics and if one of the filters is trivial. We should also support situation when filter is And(GreaterThan(port, 10), GreaterThan(port, 12)), which should result in simple GreaterThan(port, 12).

sadikovi commented 7 years ago

Currently it is assumed that Spark will handle this case and will never give us predicate like this, but when using just library it becomes quite handy to have this functionality.