At the moment none of our scanners have the ability to detect if a previously indexed document has disappeared. IIRC the old version of File Scanner that was based on directory watches did have this, but it had to be scrapped due to issues with the JDK implementation (see #130).
initial thoughts:
Build up a data structure to hold the list of Id's seen during a scan (pick an efficient one)
At the end of the scan identify any not seen during the scan, and then if the last status for that doc is not "delete" or "error" send a delete.
Serialize & persist this structure at the end of each scan.
When starting up a scanner check for and load the serialized structure
Also, make sure processor related documentation/javadocs clearly mention the possibility that the document may represent a deletion (often processors will want to ignore these documents), and make sure our provided implementations have an option to ignore deletes (or not).
At the moment none of our scanners have the ability to detect if a previously indexed document has disappeared. IIRC the old version of File Scanner that was based on directory watches did have this, but it had to be scrapped due to issues with the JDK implementation (see #130).
initial thoughts:
Also, make sure processor related documentation/javadocs clearly mention the possibility that the document may represent a deletion (often processors will want to ignore these documents), and make sure our provided implementations have an option to ignore deletes (or not).