Closed lalalune closed 3 months ago
@sshivaditya2019 I've merged this but we will need to ignore levenstein distance check if content is more than 255 characters
I updated the levenshtein distance check on the server but we'll need to add a migration. I'm going to close this out and assume we'll handle migration on discord
Embeddings cost money, and 20-30% of them are probably repeats of obvious things, like "hey whats up" (in the case of a memory embedding).
When inserting embeddings, do a check to see if there are any rows where the string is the same. Bonus points for levenstein distance check to string similarity. Then if that row returns and has a vector, use it instead of making a new vector.