Enhance Search Support

[Jotschi]

Add the following features:

Autocomplete / Autosuggestions (Search-as-you-type)

Elasticsearch recommends to use a dedicated suggester / suggester index which uses FSTs in order to achieve faster performance.

NOTE: Suggesters can't be used for our usecase since matching always starts at the beginning of the text in the index. We could only use suggester if we add another dedicated field which would be authored by the editor of the content.

One alternative would be to use an edge_ngram index:

• https://stackoverflow.com/questions/29753971/elasticsearch-completion-suggest-search-with-multiple-word-inputs
• https://qbox.io/blog/an-introduction-to-ngrams-in-elasticsearch

---

Autocomplete can be solved using the suggested terms. Autosuggest can be solved using the returned hits.

• [x] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [x] PoC
• [x] OSU

Highlighter option / NGram Prefix

{
  "query": {
    "match_phrase_prefix": {
      "fields.description": "Spa"
    }
  },
  "highlight": {
    "fields": {
      "fields.description": {
        "number_of_fragments": 2,
        "fragment_size": 5,
        "pre_tags": [
          "<strong>"
        ],
        "post_tags": [
          "</strong>"
        ]
      }
    }
  },
  "_source": {
    "exclude": [
      "*"
    ]
  }
}

Suggester Option

TBD

Pro

• Fastest autocomplete solution.
• Best control over auto completion and suggested phrases

Con

• Requires more time and effort to setup the dedicated index
• Autocomplete Index can not be directly populated via the existing content data

PDF / Attachment / Pipeline handling - #37

Pipeline handling requires the installation of the ingest plugin. This plugin has its own REST endpoint and needs to be configured separately. Attachments are stored via the regular putObject command. The action needs to reference the configured pipeline.

• 5.6

• [x] Implementation

• [x] Tests

• [x] Docs

• [x] Changelog

• [x] PoC

• [x] OSU

Did you mean

Along the query a suggest request can also be submitted in order to retrieve suggestions by elasticsearch.

The did you mean feature requires a trigram analyzer with a shingle filter.

• 2.4

• 5.6

• [x] Implementation

• [ ] Tests

• [ ] Docs

• [ ] Changelog

• [x] PoC

• [x] OSU

Partial word matching

It is possible to use the match_phrase_prefix in order to match all tokens of the query except the last. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase-prefix.html

• [x] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [x] PoC
• [x] OSU

Facets / Aggregations - #20

Aggregations can be configured within the query. The user needs to use the rawSearch endpoint in order to access the aggregated search results.

• [ ] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [ ] PoC
• [ ] OSU

Synonymes

Synonymes can be [configured within the index settings] (https://www.elastic.co/guide/en/elasticsearch/guide/current/using-synonyms.html). - #185

• [ ] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [ ] PoC
• [ ] OSU

Problem

The index configuration can currently not be customized. - #185

Stopwords

Stopwords can be configured within the index settings.

• [x] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [x] PoC
• [x] OSU

Problem

The index configuration can currently not be customized. - #185

Result-Highlighting

Add highlight configuration to query.

{
"highlight": {
    "fields": {
      "fields.description": {
        "number_of_fragments": 0
      }
    }
}

• [x] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [x] PoC
• [x] OSU

Problems

Highlight info is only accessible via searchRaw. The search raw response lacks various needed information (e.g.: node path)

• Update GraphQL API to also expose the highlight information
• Invoke an additional graphql request for each search result.

Boosting

Boosting is already supported by the ES query.

• [ ] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [ ] PoC
• [ ] OSU

Indexing of third party systems

• [ ] Implementation
• [ ] Tests
• [ ] Docs
• [ ] Changelog
• [ ] PoC
• [ ] OSU

[Jotschi]

The html field should automatically apply the htmlstrip-charfilter. https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-htmlstrip-charfilter.html

[Jotschi]

TODO

• [x] Add permission tests for new search perm handling
• [x] Add endpoint to directly expose ES API
• [x] Add code to update documents if roles are updated or permissions are changed
• [x] Add html strip char filter
• [x] Ensure that read_publish perm is correctly handled (AbstractSearchHandler#87)

[Jotschi]

Problems

How to handle Elasticsearch custom index configuration - #185

Option A (preferred)

Add settings to the Gentics Mesh schema.

Con:

• Effort to add the needed config relatively high.
• ES and Gentics Mesh configuration gets mixed up

Pro:

• The index configuration is currently created using the schema. It thus makes sense to put the index config within the schema itself.

Option B

Place the custom configuration within an external JSON file which will be used when creating new indices. The default index configuration would be merged with that file. We already do this for the .raw field handling.

Con:

• How to synchronize those template files across the cluster?

How to expose Elasticsearch aggregations / suggestions / result highlighting

Option A (preferred)

Directly expose the raw result and do not wrap the response. → already implemented via /rawSearch/ endpoints.

Con:

• Does not expose all needed information. In most cases an additional GraphQL per hit is needed to get all the needed info for the found document (e.g.: path info)

Pro:

• Does not require custom handling and is thus more resilient to API changes.

Option B

Wrap the result fields and thus make it possible to access those fields via GraphQL.

Con:

• Option may not be able to handle all custom Elasticsearch properties. Only a selective subset of features would be supportable. Each ES feature/property would require custom handling in Gentics Mesh.

Pro:

• No need to invoke multiple requests. All the needed data could directly be loaded via GraphQL.

{
  nodes(query: "ESQuery") {
     elements {
          // Returns the highlight info per hit
          highlights {
               key
               value
          }
      }
     // Returns the aggregation info
     aggregations
     // Returns the suggestion info
     suggestions
  } 
}

Option C

Expose the raw result within GraphQL.

Con:

• The raw result would only be stored as a nested escaped JSON string. The user would need to unescape the string. This appears very ugly and not elegant at all.

Pro:

• Easier to implement in comparison to option B.

{
  nodes(query: "ESQuery") {
     // Returns the full raw elasticsearch JSON
     rawHits
     elements {
          // Returns the raw elasticsearch hit JSON
          rawHit
      }
  } 
}

Option D - #187

Add an endpoint which accepts a special form of GraphQL query. The query would also contain the elasticsearch query. The GraphQL query would be used to load the needed data on a per/hit basis.

{
  hit {
      uuid
      language
  } 
}
----
{
  "query": ES_QUERY
}

The resulting elastic search response would be enhanced such that the "_source" document would be omitted and instead the GraphQL data would be added.