search

logv / sybil

columnar storage + NoSQL OLAP engine | https://logv.org
https://logv.org
Other
305 stars 26 forks source link

[query path] early filtering #97

Open okayzed opened 5 years ago

okayzed commented 5 years ago

this task is to add early filtering to the loading samples off disk. this allows us to skip over records that are not applicable instead of waiting until the query stage.

according to abadi, late materialization can speed up queries enormously, but its not clear to me how to late filter when each thread is loading a separate column off disk

okayzed commented 5 years ago

my first attempt was 1) slower than filtering after loading and 2) giving wrong answers. :cry: