search

Anthony-Nolan / Atlas

A free & open-source Donor Search Algorithm Service
GNU General Public License v3.0
9 stars 5 forks source link

searches sometimes take days to finish #682

Closed mmelchers closed 2 years ago

mmelchers commented 2 years ago

During testing of our ATLAS deployment we had several searches which ran for days. An example is: "IsRepeatSearchSet": false, "ResultsFileName": "2b2d1724-c11b-4ba5-bc42-1a39a966e569.json", "MatchingAlgorithmTime": "00:00:26.9384279", "MatchPredictionTime": "2.00:21:59.9907725", "SearchRequestId": "2b2d1724-c11b-4ba5-bc42-1a39a966e569", "MatchingAlgorithmHlaNomenclatureVersion": "3450", "BlobStorageContainerName": "atlas-search-results", "TotalResults": 8,

Some of the parameters were: shared database: GP_S_Gen5_8 standby matching database: GP_S_Gen5_1 active matching database: GP_S_Gen5_8 elastic scaling plan: EP2 HLA was provided for loci A, B, C, DRB1, DQB1 and DPB1. It was a mix of real patient HLA with varying resolutions, including LR HLA. The HLA of patients for searches that took this long was not even low resolution, but at least 2 field.

This was happening when I was performing a run of 50 patients. 34 have not finished after 68 hours. Only 8 finished within an hour of starting the search.

HAP-E and Optimas finish 48 out of those exact same 50 patients within 23 minutes and the remaining two within an hour.

This is totally unacceptable.

benbelow commented 2 years ago

@mmelchers the results of my testing against the WMDA test environment after the fixes for this issue were merged were as follows:

Additionally, I ran a search with a "theoretical worst case" for patient HLA (i.e. using XX codes at A/B/DRB1, using the first fields that are known to expand to the most possible g-groups), using the global HF set. This search did take much longer to run than the cases we saw above, but the time was in the low hours, rather than days.

I'm confident with this testing that this issue can be closed.

@mmelchers I'm going to close this issue based on this testing. If you find any more cases that take days rather than minutes or hours in your acceptance testing, let me know and we can raise a new issue to dig down further.