Technical solution for global search capabilities

[bogdanromanx]

Design the technical solution for global search capabilities (expected outcome is merely a technical design):

• a user must be able to search known data types from Fusion across all projects in different access scopes (exemplified in the mocks as "public" and "BBP only")
• search must support full text queries
• must support filtering based on the defined fields
• must support aggregation (facets) with the result counts; data types may share fields while some are specific; also some fields may have disjoint values (e.g. annotations may contain both e-type and m-type values)
• search must be able to provide suggestions during typing based on known values of specific fields
• search results must present previews as part of the result cards where available
• selecting a search result will display the available information about that data (resource view)
• search must be able to support hierarchies of values (ontologies, like for example: brain region)

General approach:

• identify the initial data types that need to be indexed (e.g.: Trace, Morphology, IonChannel etc.)
• identify their fields (e.g. name, brainRegion, e-type, m-type, preview etc.)
• derive an index configuration (mappings, settings, analyser config, shards, replicas etc.)
• setup an indexing process that collects resources from subscribed projects that match the index expectations in terms of defined fields
• later on: identify necessary transformations (graph traversals) required for augmenting the resources with missing expected fields (e.g. if the subject is defined as a separate resource)

Known issues that need to be addressed:

• the solution cannot rely on indices within a project because there's a limit on how many indices can be queried at once; the proposed alternative is to allow project "administrators" to opt-in to global search within a predefined access scope thus expressing consent of making data available for search with different access control requirements
• if the global indices are built by replaying the entire event log, how can project data be indexed if a project subscribes to global search while the indexing process is actively running (i.e. the current offset is higher than the some data in the newly subscribed project)
• how can data be removed from global search if a project unsubscribes or changes visibility
• it is expected that a subset of the data will not match the expectations of the index, the process could skip and report the issues or ignore missing fields
• as part of the user experience, when a user opts in a project to global search, there's an expectation that data will start becoming available immediately; how can this be achieved?
• if data (either additional metadata or distribution - files) is indexed into the global scope, how can a resource be previewed when access to its origin is restricted?

Initial mock-ups: https://xd.adobe.com/view/8ac356ae-0294-45d6-b2bc-c17441c9f780-df90/screen/c5d66708-878a-4be7-a097-34a9ab088c37/

Acceptance criteria:

1. Create an issue that describes the technical implementation that includes:
   • how global search and indexing work
   • new resource definitions (e.g. scope, globalview?) and their lifecycle
   • api definition
   • sdk additions
2. Create the set of first possible implementation issues with complete details.

ES Benchmark:

Data setup

Data types: 10-50 Shared properties across data types: 10 Custom properties per data type: 5 Additional properties (no facets): 10 Max distinct values per property: 100

Test setup

Cluster size: 8 Cores: 16 vcpu Memory (heap): 16 Memory (total): 32

Count of projects: 500 Number of replicas: 2 (3 shards) increasing to 4 (5 shards) after biggest test 1,000 / 1% - 6mil, 2% - 1mil, 5% - 20k, 92% - 10K docs 10,000 / 1% - 1mil docs, 5% - 100k, 94% - 10k

Example query (to be modified)

Max size per term: 100 Max size per shard: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html Track total hits: 10k

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Click to expand!

``` { "sort": [ { "_createdAt": { "order": "desc" } } ], "size": 20, "from": 0, "track_total_hits": 1000000, "query": { "bool": { "filter": { "bool": { "must": [ { "match_all": {} }, { "term": { "_deprecated": false } } ] } } } }, "aggs": { "brainLocationLabel": { "terms": { "field": "brainLocation.brainRegion.label.raw", "size": 1000000 } }, "objectOfStudyLabel": { "terms": { "field": "objectOfStudy.label.raw", "size": 1000000 } }, "type": { "terms": { "field": "@type", "size": 1000000 } }, "layer": { "terms": { "field": "brainLocation.layer.label.raw", "size": 1000000 } }, "annotation": { "terms": { "field": "annotation.hasBody.label.raw", "size": 1000000 } } } } ```

[imsdu]

Benchmarks results:

The benchmarks were only run on the 1,000-indices scenario as the 10,000 indices one hits the default limit of shards per node and goes well against elasticsearch recommendations in terms of shard sizing.

The 1,000 scenario allowed to create 90M documents imitating documents and queries we would like to achieve.

Example of document

```json { "@id": "9e1d2aee-5e20-4714-a9b4-85c38618bc20", "@type": [ "Entity", "NeuronMorphology", "Dataset", "ReconstructedCell" ], "mainType": "NeuronMorphology", "organization": "org", "project": "index", "revision": 27, "createdAt": "2017-06-19T21:17:07.485Z", "createdBy": "lily-myers", "updatedAt": "2019-12-29T18:00:51.659Z", "updatedBy": "lyndon-richards", "name": "932 326 additional", "description": "877 gad 1000 anterodorsal 607344830 932 560581559 cerebal midlines paratrochlear 538 312782648 oligodendrocyte constrained 1100 postsynaptic apwaveform 916 orientations meshes", "dateCreated": "2016-10-08T19:52:24.309Z", "license": "License 4", "subject": { "@type": "Subject", "age": 991, "species": { "label": "Rattus norvegicus" } }, "brainLocation": { "@type": "BrainLocation", "brainRegion": { "label": "Lateral dorsal nucleus of thalamus" }, "layer": "Layer 3" }, "annotation": [ { "@type": "ETypeAnnotation", "label": "POM_IN" }, { "@type": "ETypeAnnotation", "label": "EPI_TC" } ], "objectOfStudy": "paratrochlear habenular 560581559", "contribution": [ { "agent": { "name": "Aiden Brooks" }, "hadRole": { "label": "supervision role" } }, { "agent": { "name": "Clark Martin" }, "hadRole": { "label": "data collection role" } }, { "agent": { "name": "April Adams" }, "hadRole": { "label": "supervision role" } }, { "agent": { "name": "Natalie Wright" }, "hadRole": { "label": "neuron morphology reconstruction role" } }, { "agent": { "name": "Charlotte Higgins" }, "hadRole": { "label": "data collection role" } }, { "agent": { "name": "Dainton Douglas" }, "hadRole": { "label": "neuron electrophysiology recording role" } }, { "agent": { "name": "Edward Jones" }, "hadRole": { "label": "neuron electrophysiology recording role" } }, { "agent": { "name": "Rosie Payne" }, "hadRole": { "label": "neuron electrophysiology recording role" } }, { "agent": { "name": "Lily Myers" }, "hadRole": { "label": "neuron morphology reconstruction role" } }, { "agent": { "name": "Jacob Ellis" }, "hadRole": { "label": "data collection role" } } ], "generation": { "@type": "Generation", "activity": { "@type": [ "NeuronMorphologyCorrection", "Activity" ], "startedAtTime": "2019-03-23T04:35:54.592Z", "endedAtTime": "2020-11-21T12:24:49.893Z", "notes": "877 split paratrigeminal corticohypothalamic 312782652 932 levels 560581559 589508447 entorhinal" } }, "customNeuronMorphology1": "2020-08-15T14:59:35.813Z", "customNeuronMorphology2": "0", "customNeuronMorphology3": 939, "customNeuronMorphology4": "vomeronasal split 560581559 589508447 lamina", "customNeuronMorphology5": 0.9178825906360844 } ```

NB: the term nested field in this comment holds the elasticsearch meaning.

The queries (enclosed with this comment) were the following:

• simple: all documents, 4 aggregations with no nested field
• annotation: all documents, 5 aggregations, one aiming a nested field annotation
• annotation-role: all documents, 6 aggregations, two aiming a nested field annotation and role
• filter-type: documents matching a given type matching ~4,5M documents, 6 aggregations, two aiming a nested field annotation and role

Each of these queries were run against:

• all 1,000 indices (1,000 shards) and 90M documents (with and without cache),
• the 10 largest indices (10 shards) and 60M documents (with and without cache)

Each of these 16 combinations were repeated 100 times where details are enclosed in the gatling report. global-search-benchmarks.zip

Conclusions on this benchmark:

• Cache in elasticsearch really makes the difference in response times
• Querying a lot of shards has an overhead on response times and on the elasticsearch cluster (cpu/memory/threads/disk space) itself.
• Computing the exact number of matched documents adds latency, setting an upper limit reduce the response times
• Nested fields adds latency especially when we miss the cache
• Filtering on a type/project to aim a subset of documents reduce response times by a factor larger than the number of matching documents
• Adding replicas in these these tests did not improve response times but should bring value in a scenario with a lot of different concurrent searches or a scenario where one or several elasticsearch nodes go down.

[imsdu]

2608 has been completed according to benchmarks and to discussions within the team.