Store additional donor info in matching database

[zabeen]

Dependency of matching on donor store leads to the exception described here #1055

To avoid this issue, three additional fields should be added to the Donors table on matching algorithm db:

• ExternalDonorCode
• EthnicityCode
• RegistryCode

These fields should be set via:

• data refresh (via DonorReader)
• searchable donor updates (via this model)

Release Management

• new columns must be seeded before the dependency on donor store during search can be dropped
• a feature flag should be used: useDonorInfoStoredInMatchingAlgorithmDb
  • when false (default), donor info will be fetched the existing way, via call to the donor store db
  • when true, donor info stored in the new props will be used instead
  • Trace messages should be used to log which code path was followed so we can be sure the FF is in the correct state.

  • 1066 covers setting up FF infrastructure on Atlas

  • before #1066 is merged, use a bool constant as a placeholder for the FF
  • after #1066 is merged, define the new FF as a terraform resource in the donor_import module and use IFeatureManager to grab its value (value should be cached as it will be used on every search).
• the easiest way to seed the new columns is via data refresh

Release steps would thus be:

1. Release changeset (incl. db migration) with the feature flag set to false
2. Run data refresh job
3. Set feature flag to true
4. Release changeset to remove donor store dependency (this will be covered by #1067 )

[zabeen]

@mmelchers It will take a few weeks to implement, test and release this.

Could you please investigate whether it is a single org who is responsible for the mass deletes and re-creates, followed by 100s of searches that bring the deleted donors back?

If it is a single org, could you notify them of the reason for the search failures, and ask them to allow 24 hours for donor updates to propagate before running the searches? At least until the fix has been deployed to live.

[mmelchers]

@zabeen the organisations doing the large updates are in many cases not the organisations that are doing the searches. These donors may appear in searches started by organisations all over the world. We cannot ask those organisations to not start searches for 24 hours, because another organisation is performing a large update.

[zabeen]

@mmelchers understandable, but I was just wondering if it was more than a coincidence that there were so many searches failing for this reason so soon after the mass delete/recreate event.

I was hoping S&M db would show a relationship between who is kicking off the searches and who is doing the mass updates.

[zabeen]

Testing Notes

1) Dev test

• Upload a donor import file with a new donor, file should have values for external donor code, ethnicity code and registry code
• Use debug function to check that the donor has been added to active matching db with the three correct code values

2) Regression test

• Run a search and check that:
  • ExternalDonorCode, EthnicityCode and RegistryCode are being populated for each matching donor result.
  • The AI logs for the search include a trace message recording which data source was used to retrieve the donor info (i.e., Atlas donor store or Matching Algorithm db).
• To be run:
  • before data refresh, with feature flag set to false
  • after data refresh, with feature flag set to true

3) Bug fix test

To be run with feature flag set to true

Part 1:

• Pause the donor management function
• Upload a donor import file with a "delete" update
• Run a search that will bring back the deleted donor (i.e., patient has same HLA as the deleted donor)
• Outcome: the search should complete successfully, and report the deleted donor as a match

Part 2:

• Restart the donor management function
• Check the delete update has been applied to the matching algorithm db (i.e., donor should be marked as unavailable for search)
• Re-submit the search request from Part 1
• Outcome: the search should complete successfully, and the deleted donor should no longer be a match

[zabeen]

Dev work done, but release and testing blocked by #1066

Update: #1066 completed, ticket is ready for testing

[DmitriyShcherbina]

@zabeen Hi! From my perspective, testing status: Ok

e93a6917-455d-4489-a437-46bc542ff9f5 - after DEV-ATLAS-MATCHING-DONOR-MANAGEMENT-FUNCTION pause and upload donor import file with 'delete' donors list

8dba3c96-2a32-4705-9b4d-3bec1daa5ce9 - after DEV-ATLAS-MATCHING-DONOR-MANAGEMENT-FUNCTION restart

[zabeen]

Release to WMDA as of 10th October 2023

• Code changes and data migrations have been released
• Data refresh for v3.54.0 should be taking place this weekend
• Once that completes, the feature flag can be enabled

[zabeen]

WMDA release completed, see #1066 for release notes.