search

lanterndata / lantern

PostgreSQL vector database extension for building AI applications
https://lantern.dev
GNU Affero General Public License v3.0
790 stars 57 forks source link

Infer array dimensions from index expressions #175

Closed var77 closed 1 year ago

var77 commented 1 year ago

Description

When index is being created via expression and dim option is not passed we can not infer the array dimensions from the table row. So by this PR we will check if the expression list is not empty when inferring the dimensions, we will evaluate the expression over the first row of the table and try to get the array dimensions from the result.

Relates to #174 Issue #162

github-actions[bot] commented 1 year ago

Benchmarks

metric old new pct change
recall (after create) 0.740 0.740 -
recall (after insert) 0.742 0.764 +2.96%
select bulk tps 218.594 228.773 +4.66%
select bulk latency (ms) 34.397 33.716 -1.98%
select bulk latency (stddev ms) 11.556 9.489 -17.89%
create latency (ms) 1570.497 1456.375 -7.27%
insert bulk tps 13.726 14.685 +6.99%
insert bulk latency (ms) 72.843 68.082 -6.54%
insert bulk latency (stddev ms) 3.065 2.094 -31.68%
disk usage (bytes) 6348800.000 6348800.000 -
var77 commented 1 year ago

Currently after creating an index with expression, we are not able to query it using <-> operator as the operator usage check fails. I have described the issue in this comment . Should we fix the issue in this PR or work on it in a different PR?

cc: @Ngalstyan4 @dqii

codecov[bot] commented 1 year ago

Codecov Report

Merging #175 (311b985) into main (e87563a) will decrease coverage by 0.14%. Report is 1 commits behind head on main. The diff coverage is 86.66%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #175 +/- ## ========================================== - Coverage 83.06% 82.93% -0.14% ========================================== Files 17 17 Lines 1175 1201 +26 Branches 251 256 +5 ========================================== + Hits 976 996 +20 - Misses 85 87 +2 - Partials 114 118 +4 ``` | [Files](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata) | Coverage Δ | | |---|---|---| | [src/hnsw/insert.c](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata#diff-c3JjL2huc3cvaW5zZXJ0LmM=) | `82.25% <100.00%> (ø)` | | | [src/hooks/executor\_start.c](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata#diff-c3JjL2hvb2tzL2V4ZWN1dG9yX3N0YXJ0LmM=) | `89.47% <100.00%> (ø)` | | | [src/hooks/post\_parse.c](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata#diff-c3JjL2hvb2tzL3Bvc3RfcGFyc2UuYw==) | `88.60% <100.00%> (-1.27%)` | :arrow_down: | | [src/hnsw.c](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata#diff-c3JjL2huc3cuYw==) | `80.00% <88.88%> (+0.22%)` | :arrow_up: | | [src/hnsw/build.c](https://app.codecov.io/gh/lanterndata/lantern/pull/175?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lanterndata#diff-c3JjL2huc3cvYnVpbGQuYw==) | `81.11% <84.84%> (-0.30%)` | :arrow_down: |