search

Stevenic / vectra

Vectra is a local vector database for Node.js with features similar to pinecone but built using local files.
MIT License
321 stars 29 forks source link

Error saving index: RangeError: Invalid string length #38

Open Gitter-MI opened 6 months ago

Gitter-MI commented 6 months ago

Hi Steven, vb here from the openai community forum. When creating an index I ran into this error where the filename is too long which causes the index creation to fail.

Error: Error adding document "C:\Users\VB\source\repos\evals.git\lfs\objects\03\5f\035fbfc20caee391612ae6e8f394b0788406f493b208e67419bb1eea17f9575c": Error: Error saving index: RangeError: Invalid string length at LocalDocumentIndex. (C:\Users\VeitB\AppData\Roaming\npm\node_modules\@stevenic\codepilot\node_modules\vectra\lib\LocalDocumentIndex.js:258:23) at Generator.throw () at rejected (C:\Users\VB\AppData\Roaming\npm\node_modules\@stevenic\codepilot\node_modules\vectra\lib\LocalDocumentIndex.js:29:65)

This is the filename "035fbfc20caee391612ae6e8f394b0788406f493b208e67419bb1eea17f9575c". The file has no extension.

Ps. I know I should have excluded the .git folder anyways.

Gitter-MI commented 6 months ago

I tried to add the same project (OpenAI evals) again to vectra using Codepilot and this time made sure to exclude the directory and files that previously caused the error. But the same error was thrown later in the process. I have come to believe that this may actually be an issue with rate limiting on the side of the OpenAI account used. In this particular case the account was Tier 1 with a token limit of 1.000.000 TPM and that the error message does not correctly decribe the reason for the error. If this assumption is correct then in this scenario it is not possible to create a project because the only way to rebuild is to create from scratch and the same error will occur again.

Stevenic commented 6 months ago

Yeah at some point the JSON just gets to be too big... The limit for an index is around 2,000 files currently. I'm hoping to address that in Vectra 2 with a more compact file format.