search

spiraldb / vortex

An extensible, state-of-the-art columnar file format
https://vortex.dev
Apache License 2.0
1.01k stars 28 forks source link

Use array len as denominator for selectivity #1468

Closed robert3005 closed 6 days ago

robert3005 commented 6 days ago

True range is too expensive of a metric to calculate given current apis

robert3005 commented 6 days ago

I have found this while spot checking profiles - in q19 computing true range was 30% of all our compute time

gatesn commented 6 days ago

I can maybe believe it, but the benchmark run doesn't agree with you?

robert3005 commented 6 days ago

It’s overall 5% better or am I looking at wrong numbers?

danking commented 6 days ago

The q19 row didn't show an improvement which seems suspicious given the profile

robert3005 commented 6 days ago

Yeah so I have run this on top of #1466 on my laptop and that was an improvement. I imagine our ci machines would produce a different profile. Could also be that this is bigger improvement with #1466 that it is standalone