lanterndata / lantern

PostgreSQL vector database extension for building AI applications
https://lantern.dev
Other
755 stars 53 forks source link

Use proper index selectivity in hnswcostestimate function #279

Closed therealdarkknight closed 8 months ago

therealdarkknight commented 8 months ago

As per postgres documentation, indexSelectivity is the estimated fraction of parent table rows that will be retrieved during the index scan. This was previously hardcoded to 1, but it should instead be the fraction of parent table rows that are in our index (which can be less than 1 when we have partial indexes). The genericcostestimate function that we call appears to compute this value for us, and so we can use that instead.

I also added tests in hnsw_cost_estimate that compute the indexSelectivity values (shown in the lantern debug logs that are outputted during this test) for partial indexes, to make sure that indexSelectivity is computed properly.