sennetconsortium / entity-api

A set of web service calls to return information about SenNet entities
MIT License
0 stars 0 forks source link

New features oriented to improve index procedure efficiency #396

Closed maxsibilla closed 1 month ago

maxsibilla commented 1 month ago

During indexing, all we need are the node properties from Neo4j directly AND a few on_read_trigger generated ones. But the current GET /entitites/<id> returns ALL trigger generated properties, totally inefficient and unnecessary. And the index procedure has to remove the ones that are not specified in the mapping json with lots of for loops, such repetitive work is a total waste of time and further reduces the efficiency.

To address the above issues in the entity-api,

Afterwards, create a specialized endpoint (based on the current GET /entities/<id> but with no property filtering needed): GET /documents/<id> which returns a subset of the regular entity json to be used for index, without including the following generated fields (not used by index process):

AFTER the new features get tested, switch to use this new endpoint in search-api: https://github.com/hubmapconsortium/search-api/issues/756

Related HM card: https://github.com/hubmapconsortium/entity-api/issues/630 Related PRs: https://github.com/hubmapconsortium/entity-api/commit/2b34a0fcc645b9a4e7126684d4e331eaee78a1c4#diff-cd8a3bff2418f86ca995d983b9e0d2ed942f3a2a10b8eb393b15f18f56ea3fb4 https://github.com/hubmapconsortium/entity-api/pull/696/files#diff-04791d82dd15fdd480f084d7ef65a10789fa5012cb7935f76080763444d48a00

maxsibilla commented 1 month ago

This is directly related to #334