fulmicoton
commented
1 month ago Would this approach violate any internal implementation contracts of Tantivy, and is it feasible?
It will most likely not work yes.
The problem has to do with the delete and commit work. Deletes are performed right before serialization.
The reason something like
- add_doc(1)
- delete_doc(1)
- delete_doc(2)
- add_doc(2)work the way you expect, is because we attach an opstamp to each document and each delete operation, to know in which order those operations happened.
With your scheme, two concurrent writes could end up with very different outcomes.
- add_doc(1)
- add_doc(2) - delete_doc(1)
- delete_doc(2)You could end up with no docs, doc1, doc2, doc1 and doc2 in the resulting tantivy index.
It will NOT look like the transaction were executed in the order of them taking the write lock.
cyccbxhl
I'm considering an integration of Tantivy with PostgreSQL, which has a multi-process architecture where each insert/update/delete/vacuum operation is handled by a separate process. However, Tantivy's IndexWriter cannot be initialized simultaneously by multiple processes in the same directory due to the INDEX_WRITER_LOCK.
To enable concurrent insert/delete operations, I'm thinking of removing the INDEX_WRITER_LOCK when initializing the tantivy::IndexWriter within the PostgreSQL insert backend process. Instead, I plan to introduce a read-write lock located in shared memory. This lock would be acquired as a read lock during add_document/delete_term operations and as a write lock during commit operations (the garbage collection process would also acquire a write lock).
Would this approach violate any internal implementation contracts of Tantivy, and is it feasible?
I'm looking for guidance on whether this strategy could potentially disrupt Tantivy's internal mechanisms and whether it aligns with Tantivy's design principles.