Narrative Service: Rework Query Engine & Provenance feature

[ir0ntr0nik]

The idea is to replace the current provenance retrieval based on precomputed predications with the computation at runtime. The following things have to be modified:

1. Change the current route of getting provenance information to the runtime variant. (and expect equal results)
2. Reduce the complexity of the PredicationInvertedIndex
3. Adjust the query engine (query_inverted_index_for_fact_pattern & process_query_with_expansion)
4. Add a utility module that provides efficient joins (1) and intersections (2) of two lists of integers (as doc_ids)

[ir0ntr0nik]

SQL statement to rename the column provenance_mapping to document_ids:

ALTER TABLE predication_inverted_index
RENAME COLUMN provenance_mapping TO document_ids

Example for a document_ids entry: [id_1, id_2, id_3, ..., id_n] (list stored as a string, ids are always integers)

[ir0ntr0nik]

python compute_reverse_index_predication.py sidenotes:

• time of execution ~3h
• the size of the python process is ~110GB (at max)
• the size of the materialized table after VACUUM is ~15GB

[HermannKroll]

Solved.

[HermannKroll]

Can we improve the error shown to users when explanation has failed? For instance, "Explanation failed - please try again"

[HermannKroll]

Sort by confidence descending (in frontend)