Fix bug in handling of soft deleted documents when updating settings

[loiclec]

Pull Request

Related issue

Fixes (partially, until merged into meilisearch) https://github.com/meilisearch/meilisearch/issues/3021

What does this PR do?

This PR fixes the bug where a missing key in documents database internal error message could appear when indexing documents.

When updating the settings, before clearing the database and before creating the transform output, we now modify the ExternalDocumentsIds structure to get rid of all references to soft deleted document ids in its FSTs.

It used to be that updating the settings would clear the soft-deleted document ids, but keep the original ExternalDocumentsIds structure. As a consequence of this, when processing a future document addition, we could wrongly believe that a document was being replaced when, in fact, it was a completely new document. See the tests bug_3021_first, bug_3021_second, and bug_3021 for a minimal test case that would have reproduced the issue.

We need to take special care to:

• evaluate how users should update to v0.30.1 (containing this fix): dump? reimporting all documents from scratch?
• understand IF/HOW this bug could have caused duplicate documents to be returned
• and evaluate the correctness of the fix, of course :)

[Kerollmops]

Let's merge this! bors merge

[bors[bot]]

Build succeeded:

• Run Clippy
• Run Rustfmt
• Tests on macos-latest
• Tests on ubuntu-20.04
• Tests on windows-latest