Open FroMage opened 2 weeks ago
cc @jmartisk
Quickly updating the corresponding embeddings if one of the source documents changes is unfortunately not trivial, because there's no 'equality' operator in this world. So, to keep track of which embeddings are related to which document, you probably need to use some sort of ID. That ID can be part of the metadata.
Could you perhaps, in the PrePersist
method, take the ID of the document that is being updated, do something like
// remove the original embeddings
embeddingStore.removeAll(new IsEqualTo("id" , talk.id));
// create the new embedding
Map<String, String> metadata = new HashMap<>();
metadata.put("title", talk.title);
metadata.put("id", talk.id.toString());
Document document = new Document("Title: "+talk.title+"\nID: "+talk.id+"\nDescription: "+talk.description, Metadata.from(metadata));
List<TextSegment> split = DocumentSplitters.recursive(2000, 0).split(document);
List<Embedding> newEmbeddings = embeddingModel.embedAll(split).content();
store.addAll(newEmbeddings);
This also depends on the particualar embedding store implementing the removeAll
method though, and I think PgVector doesn't support it right now :/
I've just noticed there's also the related https://github.com/langchain4j/langchain4j/issues/1299 (except it's about using the embedding ID instead of one stored as part of the metadata)
PgVectorEmbeddingStore
supports removeAll(Filter)
where Filter
can be metadataKey("id").isEqualTo(talk.id)
The current RAG model for pgvector is to store the documents in their own table.
In my application my source documents already have a table:
So, when I iterate those to index them, they all go in their separate table:
This leads me to wonder how I can keep my model and the index in sync. What do I do when I update a single
Talk
entity? Do I need to re-index the entire store?Intuitively, I was expecting to be able to do something like:
But I'm not too sure how to wire this up.
I suppose that a
PanacheEmbeddingStore
could have this sort of API for batch reindex, given this: