search

Stevenic / vectra

Vectra is a local vector database for Node.js with features similar to pinecone but built using local files.
MIT License
321 stars 29 forks source link

Sloppy breaks certain strict-only use-cases. #30

Open steveruizok opened 7 months ago

steveruizok commented 7 months ago

First off, great library! We're using it to do our "AI powered" docs search on tldraw.dev.

Some of the code in this library depends on turndown, which in turn depends on sloppy.js. Certain "strict mode only" contexts, such as server-side routes on our Next.js app, will throw when this code and its sloppy with usage is present. To fix this, I've had to patch out the exports of the WebFetcher and other items in the module's index.js file.

I'm not sure what the right fix is here, however you could consider separate export entry points, i.e. import { WebFetcher } from "vectra/WebFetcher" so that the extra exports are not pulled in when importing from vectra alone?

Stevenic commented 7 months ago

Good catch. I may break out the CLI piece to a separate package. I've started thinking about a number of changes for the next version of Vectra and will start a discussion next week. I'm probably going to put a wrap on the current bits and call it 1.0 and then start working on version 2.0. I want to change the file format to something more compact then the current JSON format for index.json and I'll probably move away from folders to a new <index>.vectra file format. I've worked out how to make managing larger indexes really efficient.

Stevenic commented 7 months ago

The cool thing about Vectra 2's file format is that you'll be able to merge document indexes by simply concatenating .vectra files. That means you can create per document indexes and then merge all of the .vectra files for a given set of documents into a single .vectra file that lets you search over the set.