search

lanterndata / lantern

PostgreSQL vector database extension for building AI applications
https://lantern.dev
GNU Affero General Public License v3.0
790 stars 56 forks source link

Add binary versioning #249

Closed ezra-varady closed 11 months ago

ezra-varady commented 11 months ago

This addresses #239. It generates a version header that gets included in the binary and compared to the SQL API version. If they do not match it will refuse to interact with the index. Unfortunately the system catalog does not make extension information in C afaict, so we execute an actual query to get the SQL API version. The result is cached to minimize the overhead but this means that once the mismatch has been resolved the connection must be reset. I've tested it locally and it behaves correctly.

I also added a new class of tests to check that errors are generated during update tests. I don't love the setup, so if anyone has ideas please let me know. I added a new class of tests, test-misc that are intended for use in situations like this where we want to call a single test in a very specific instance. I couldn't reuse the parallel test because the expected output has to be different which was unfortunate. In addition to this I replaced various duplicated utils with symlinks

ezra-varady commented 11 months ago

it looks like benchmark is maybe checking out a tag without altering the version. The misc tests can't be used without bumping the version which is a bit annoying, I'll look into this tomorrow. Not sure what the deal with the missing gettext header is on osx either, it looks like it's installed?

ezra-varady commented 11 months ago

I haven't been able to recreate the issue with the benchmark runner locally in the dev container. I'm not sure what's going on exactly since this error shouldn't be possible without an update, and there is no update.

Update: I added an upterm session, when I log in the version in sql is 0.0.10 which is mismatched, but if I drop the extension and manually rerun the same commands everything works fine. Not sure what to make of this

ezra-varady commented 11 months ago

It looks like maybe it was a caching issue? The test has started passing since I added and then removed the upterm job