search

VedaWebProject / Tekst

A collaborative research platform for aligning, enriching and exploring resources on natural language texts
https://vedawebproject.github.io/Tekst/
GNU Affero General Public License v3.0
8 stars 1 forks source link

Exposed to ES mapping explosion #266

Closed bkis closed 2 months ago

bkis commented 4 months ago

The way we index annotations might open the app up to a potential mapping explosion.

We should evaluate mapping annotations as [{"k": "...", "v": "..."}, ...] instead of {k: v, ...} – especially building queries might become tricky.

bkis commented 4 months ago

These seem to be the possible ways to go about this, from most to least horrifying in terms of dev effort:

  1. Use multiple indexes (no! getting results ranking to work will be a hacky nightmare!)
  2. Find another way to index the annotation data (needs some research and evaluation, might be a lot of work, unlikely to work without losing some features such as query string queries on annotation values, would be the cleanest solution if there is a way at all)
  3. Just live with this limitation, document it, make the admin UI warn the admins if this limit is reached (can the ES API tell us?) and maybe even make the indexing of non-published resources configurable on dev-ops level with a default of false
  4. Delete the repo and look for a different carreer

Currently, solution 3 looks like the most attractive one.

bkis commented 2 months ago

Solved by #331