lanterndata / lantern

PostgreSQL vector database extension for building AI applications
https://lantern.dev
GNU Affero General Public License v3.0
790 stars 57 forks source link

remove hard limit on the number of items returned when an index is queried #78

Closed ezra-varady closed 1 year ago

ezra-varady commented 1 year ago

This should address #74 It includes a regression test where ...LIMIT 10 (the new default) and ...LIMIT 50 are tested.

Resolves #74

ezra-varady commented 1 year ago

to summarize,

I still need to add a test for the query being bigger than the table. Since LIMIT only imposes an upper bound I'm not totally sure how to induce this actually, but I'll work on it this afternoon. The case of k greater than the limit is tested and behaves correctly.

Ngalstyan4 commented 1 year ago

Thanks for the prompt fixes, @ezra-varady!

I still need to add a test for the query being bigger than the table.

I am not sure what you want to test here. But if the problem is that the table has more rows than you need, you can place the test in a transaction, drop all the extra rows in the transaction, then run the query, then rollback the transaction. See an example of this kind of thing here.

Note that you also do not need a large table to force index usage in tests. you can just disable sequential scan.

ezra-varady commented 1 year ago

Thanks, I didn't know that! I reformatted the tests in light of this. I also added a check to avoid a potential extra query I had neglected. I think I had misunderstood your suggestion w.r.t. the additional test. Please let me know if you have other suggestions

ezra-varady commented 1 year ago

removed the hanging comment and changed the log level for the tests