search

Anthony-Nolan / Atlas

A free & open-source Donor Search Algorithm Service
GNU General Public License v3.0
9 stars 5 forks source link

Batch matching results by donor metadata and HLA type #916

Closed zabeen closed 1 year ago

zabeen commented 1 year ago

This ticket is dependent on changes made as part of https://github.com/Anthony-Nolan/Atlas/issues/897.

Search orchestrator downloads all matching results, and then groups donors by HLA tying and metadata to generate match prediction requests, as an optimisation to minimise the total number of requests.

Where results batching is enabled, if matching results are also batched in the same way (by donor typing and metadata), then result blobs would not have to be all downloaded to memory before match prediction requests are generated, which would flatten the memory usage curve.

zabeen commented 1 year ago

Blocked by merge of #897, and also by performance testing to see if this change is actually needed.

zabeen commented 1 year ago

There is a better way to deal with the problem of downloading large result sets prior to match prediction. Will explain on another ticket when I get back from A/L, but closing this for now. Also, testing shows memory exception occurring later in search (#931) so the priority is to fix that step.