hubmapconsortium / search-api

HuBMAP search service and associated pieces to create an index
https://search.api.hubmapconsortium.org
MIT License
2 stars 2 forks source link

Design scalability improvements (search) #867

Open shirey opened 2 weeks ago

shirey commented 2 weeks ago

Design scalability improvements with input from IU and Harvard.

T9 MAJOR ACTIVITIES (for all milestones in T9, not just this milestone)

Update Elasticsearch indices and Search API to fix current issues with scalability

OUTCOMES (for all milestones in T9, not just this milestone)

We will refactor the Search API to improve the efficiency when the indices are being reindexed by allowing for incremental updates instead of having to replace everything as well as analyzing current procedures and improving the performance of these procedures when possible. Additionally we analyze the number of attributes that we track to remove those that aren’t needed and move those that aren’t needed in the main index store to a secondary index.

### Tasks
- [ ] https://github.com/hubmapconsortium/search-api/issues/781
- [ ] https://github.com/hubmapconsortium/search-api/issues/843
- [ ] https://github.com/hubmapconsortium/entity-api/issues/695
- [ ] https://github.com/hubmapconsortium/entity-api/issues/685
- [ ] https://github.com/hubmapconsortium/search-api/issues/841
- [ ] https://github.com/x-atlas-consortia/elastic-analysis/issues/1