Currently, we can only combine two or more segments into one another, which although works, can make our index inefficient.
We should create a directory that can read from the segment, and split it out into a temp directory (this can be done with the writer directory) then tell tantivy to process all of the deletes marked within the index (providing it is safe to do so, see below) and then re-export the directory to a new segment.
Issue notes
Deletes are not attached to the index that actually contains the documents intending to be deleted so we cannot blindly remove deletes.
Some deletes may occur after a new document has been inserted and should not affect the new documents.
Currently, we can only combine two or more segments into one another, which although works, can make our index inefficient.
We should create a directory that can read from the segment, and split it out into a temp directory (this can be done with the writer directory) then tell tantivy to process all of the deletes marked within the index (providing it is safe to do so, see below) and then re-export the directory to a new segment.
Issue notes