During indexing, all we need are the node properties from Neo4j directly AND a few on_read_trigger generated ones. But the current GET /entitites/<id> returns ALL trigger generated properties, totally inefficient and unnecessary. And the index procedure has to remove the ones that are not specified in the mapping json with lots of for loops, such repetitive work is a total waste of time and further reduces the efficiency.
To address the above issues in the entity-api,
Introduce a new trigger type: on_index_trigger, which has the same trigger method as regular on_read_trigger.
Add a specialized schema_manager method get_complete_document_result() which is very similar to get_complete_entity_result() (for now do NOT modify this method), but uses on_index_trigger instead.
For caching, use a different prefix on those index data, cache_key = f'{_memcached_prefix}_complete_index_{entity_uuid}'
Introduce a new flag in schema yaml indexed: false (default is true) for indexing purposes. This allows us to remove the use of json mapping in search-api. When exposed: false there's no need to check this indexed: false since it's a field won't be exposed by entity-api. When there's only indexed: false it means this field still gets returned to the regular GET call, but we won't index this field.
Add a specialized schema_manager method normalize_document_result_for_response() based on the current normalize_entity_result_for_response() and integrate with the indexed flag.
Afterwards, create a specialized endpoint (based on the current GET /entities/<id> but with no property filtering needed): GET /documents/<id> which returns a subset of the regular entity json to be used for index, without including the following generated fields (not used by index process):
During indexing, all we need are the node properties from Neo4j directly AND a few
on_read_trigger
generated ones. But the currentGET /entitites/<id>
returns ALL trigger generated properties, totally inefficient and unnecessary. And the index procedure has to remove the ones that are not specified in the mapping json with lots offor
loops, such repetitive work is a total waste of time and further reduces the efficiency.To address the above issues in the entity-api,
on_index_trigger
, which has the same trigger method as regularon_read_trigger
.get_complete_document_result()
which is very similar toget_complete_entity_result()
(for now do NOT modify this method), but useson_index_trigger
instead.cache_key = f'{_memcached_prefix}_complete_index_{entity_uuid}'
indexed: false
(default is true) for indexing purposes. This allows us to remove the use of json mapping in search-api. Whenexposed: false
there's no need to check thisindexed: false
since it's a field won't be exposed by entity-api. When there's onlyindexed: false
it means this field still gets returned to the regular GET call, but we won't index this field.normalize_document_result_for_response()
based on the currentnormalize_entity_result_for_response()
and integrate with theindexed
flag.Afterwards, create a specialized endpoint (based on the current
GET /entities/<id>
but with no property filtering needed):GET /documents/<id>
which returns a subset of the regular entity json to be used for index, without including the following generated fields (not used by index process):direct_ancestor
collections
,upload
,direct_ancestors
,local_directory_rel_path
(still need this field to be indexed)associated_collection
AFTER the new features get tested, switch to use this new endpoint in search-api: https://github.com/hubmapconsortium/search-api/issues/756
Related HM card: https://github.com/hubmapconsortium/entity-api/issues/630 Related PRs: https://github.com/hubmapconsortium/entity-api/commit/2b34a0fcc645b9a4e7126684d4e331eaee78a1c4#diff-cd8a3bff2418f86ca995d983b9e0d2ed942f3a2a10b8eb393b15f18f56ea3fb4 https://github.com/hubmapconsortium/entity-api/pull/696/files#diff-04791d82dd15fdd480f084d7ef65a10789fa5012cb7935f76080763444d48a00