search

blugelabs / bluge

indexing library for Go
Apache License 2.0
1.88k stars 122 forks source link

Real-time reader does not observe changes to the index #52

Closed zmb3 closed 3 years ago

zmb3 commented 3 years ago

Hi there! Coming from bleve and was surprised by the following behavior:

If I open a writer, and then simultaneously open a "near real-time reader" (with writer.Reader()) and start writing to the index, searches via the reader are unable to pick up results for anything that is written after the reader was opened. Is this the expected behavior or should the reader be able to observe future updates?

If this is the expected behavior, is it more common to keep a long-lived reader open and swap it out for a new one when the index is updated, or is the best practice to acquire a reader only when you need to perform a search and then discard it?

mschoch commented 3 years ago

This is expected, as all readers in Bluge present a stable snapshot of the index. The near-realtime readers simply have the ability to search data not yet persisted to disk, whereas the traditional readers operate on snapshots persisted to disk.

As for the best practice, generally we recommend you open up a Reader when you need to execute a search, and then Close it. This releases that snapshot so that it can be cleaned up. Holding a reader open for a long period of time would otherwise interfere with regular maintenance of the index.

zmb3 commented 3 years ago

Thanks Marty, makes sense!