Searcher.search intermittent bug after IndexUpdater.persist_to_disk

[jessiejuachon]

Searcher.search produces different results after IndexUpdater.persist_to_disk

1. Call Searcher.search on an index. Notice the returned passage ids
2. Call IndexUpdater.add, remove and persist_to_disk several times for the same set of passages (no changes to content, just new pids are removed/generated for the same passages). See that the Index files have been updated.
3. Call Searcher.search with the same query that was passed in step 1. Notice that it correctly returns the most recently inserted PID from step 2.
4. Restart application (or re-initialize the Searcher and IndexUpdater, passing the same path to the now updated index).
5. Call Searcher.search with the same query that was passed in step 1. Notice that it correctly returns the same result as in step 3.
6. Call IndexUpdater.add once for the same set of passages in step 2
7. Call Searcher.search with the same query that was passed in steps 1 and 3.

ERROR/S:

• Result is incorrect set of passage IDs. Succeeding calls to IndexUpdater.add and persist_to_disk for the same set of passages results to the same erroneous set.
• Eventually, the following error is thrown: Assertion failed: (code < maxsim_args->ncentroids), function maxsim, file filter_pids.cpp, line 47.

Note: If step 4 is skipped, then there is no error.

Test environment: CPU

[TakshPanchal]

I think the issue is IndexUpdater.persist_to_disk updates only embedding vectors. Colbert's index folder has collection.json file in which all the docs are saved. IndexUpdater.persist_to_disk should also update those collections, after updating, the index searcher should be updated with the latest collection.

[pydv9991]

any update on this issue?