Data corrupted when using fast_serach mode: "Failed to open field \"title\"'s term dictionary in the compos (truncated...)"

[keywan-ghadami]

Using master/0.9 beta and a INDEX with "use_fast_fuzzy": true results in corrupted data.

minimum example

1. create index with use_fast_fuzzy set to true:

{
    "override_if_exists": true,
    "index": {
        "name": "products",
        "storage_type": "tempdir",
        "fields": {
            "title": {
                "type": "text",
                "stored": true
            }
        },
        "search_fields": [],
        "boost_fields": {},
        "reader_threads": 1,
        "max_concurrency": 1,
        "writer_buffer": 300000,
        "writer_threads": 1,
        "set_conjunction_by_default": false,
        "use_fast_fuzzy": true,
        "strip_stop_words": false,
        "auto_commit": 0
    }
} 

2. send a document missing one of the defined fields, or having the vaule a empty sting or "-" or "_"

e.g: POST /indexes/products/documents with body {"title":""}

3.

POST /indexes/products/commit I get a error message: {"status":400,"data":"Data corrupted: 'Data corruption: : Failed to open field \"title\"'s term dictionary in the compos (truncated...)

workaround / recover

to recover from this I rebuild the index and add the document with a dummy value or I rebuild the index without using "use_fast_fuzzy".

Use case

The example above is not the real use case, usually titles on all my documents are set but they have some optional fields that are optional.

[ChillFish8]

Thanks for the report!

This is a slightly misleading error, what's actually happening here is Tantivy isn't creating the term dictionary because so far it hasn't got any terms.

I have a fix in the works for this 👌

[ChillFish8]

Didnt mean to auto close this, if you can check that the fix on master has corrected the behaviour 👌

[keywan-ghadami-oxid]

Awesome, it works! Thank you for your quick support. ❤️ .