Block vectors

[kristenbrann]

Embeddings now at the file AND block level

This updates the database to store embeddings on files AND embeddings on chunks of text within the files. Each database entry now looks like this:

{
    md5hash: string;
    embedding: Vector;
    chunks: [{
    contents: string;
    embedding: Vector;
    }]
}

Check if files have changed

Stores md5 hash on the entry and uses that to compare on open to see if existing files need to be reindexed.

Batch embeddings

In the indexing function, call the embeddings api with multiple texts at once. Saves greatly on time!

Screenshot 🤭

Misc

• throttler around embeddings call
• exponential backoff retries for openai calls
• tokenizer for determining request length on block embeddings (not yet in use for all calls)
• tested and indexing, askchatgpt, and summarize all still work -> except answers are now even better
• increased context text provided in askchatgpt prompt
• changed askchatgpt prompt for better success