Fetch donor information in matching algorithm to avoid MPA fetching donors again

[benbelow]

The matching algorithm already needs to query the donor store to fetch e.g. external donor code and registry info.

Match prediction algorithm also queries the donor store for registry + ethnicity info - but if this was included in the matching results, we could remove the MPA -> Donor SQL queries, and save some time.

[zabeen]

Adding this ticket to backlog as it would also prevent search failure caused by a matching donor that has been removed from the donor store since the search began, as was the cause of #1030

[zabeen]

HLD

Matching algorithm changes

• Extend Matching algorithm result with:
  • ExternalDonorCode
  • EthnicityCode
  • RegistryCode
• Recommend placing all these details, along with existing DonorType, into a new model: MatchingDonorInfo
  • then mark MatchingAlgorithmResult.DonorType as obsolete (do not delete as it is a breaking change)
• Set these new properties before results are written out (i.e., after this step, when donor info has been selected out of the donor store for results filtering). - If a match result donor cannot be found in the donor store, then code should not throw an exception but simply remove that donor from the result set before it is written

Match prediction changes

• remove calls to IDonorReader from activity functions, their dependencies, and from service configuration.
• get donor info needed for creating match prediction batches and for combining results from matching results instead.

[zabeen]

• If a match result donor cannot be found in the donor store, then code should not throw an exception but simply remove that donor from the result set before it is written

@IgorKupreychik I just found this in the support README. It makes a good point that if a "donor not found" exception is thrown during matching, then retry logic will automatically re-run the search by which time the donor deletion update should have reached the matching db. That's probably why we have not observed this issue in matching all this time.

On that basis, I suggest we don't remove the donor from the result set, as that would mask syncing issues between donor store and matching db.

I do think we should catch and log a clear trace message (and then rethrow) so it is easy to understand what went wrong

[zabeen]

Testing Notes

• Run a search that is known to return donors, with match prediction ON

  • Search should complete successfully
  • ExternalDonorCode should be populated for every donor object in the results
  • Most donors should be represented (i.e., have match probability values and isDonorUnrepresented should be false)

• Open the matching results file for the search request in the container matching-algorithm-results

  • For each match result:
  • DonorType should still be populated
  • New object, MatchingDonorInfo, should be present and populated

[DmitriyShcherbina]

@zabeen Testing status: Ok