Improve tracking of completed donor imports

zabeen commented 1 year ago

On completion of donor import, a message is currently sent to either notifications topic (on success) or alerts topic (for failures) - the target of these messages are the support team.

New topic

Need to add a new service bus topic, donor-import-results, that can notify the consumer service (i.e., search and match) that a donor import has completed, and its status.

Message properties:

string FileName - name of the donor import file
bool WasSuccessful - result of the upload
int ImportedDonorCount - total number of donors from file that were successfully imported
int FailedDonorCount - total number of donors that failed to be imported from the file

New topic should be terraformed, with an audit subscription.

Redirect dead-lettered requests

Further, add a new dead-letter queue listener function to forward dead-lettered import requests to the new results topic, with "WasSuccessful": false

Stop success notifications

Once the new topic is available, success messages sent to the notifications topic will become redundant and should no longer be sent out. However, message to alerts topic on failure should remain.

Switching off success notifications should be safe and non-breaking. They are already switched off in AN live, as too many messages are published due to the frequency of updates.

In addition to the code that sends out the success notification, there is an app setting set by a terraform release var that toggles publishing of these notifications - the app setting and var should be removed.

Will raise a new ticket if further failure details are required in this message, for now, all failures are logged to AI, so a support agent can determine the cause of failure that way.

zabeen commented 1 year ago

Extra dev notes from discussions with @disaev-nmdp : Q: should I get inserted/failed donors count from DonorImportHistory table? Or just simple message with WasSuccessful: false without any numbers?

Answer: I am not sure how useful the counts are for unsuccessful imports... but as we do import from file in batches, it's possible some updates did get processed before a problem occurs.

I think it would be useful to add a new prop, enum ImportFailureReason, with options of RequestDeadlettered and ErrorDuringImport (this can be a catch all for any of the exceptions that can be thrown during import). We could add a new prop to hold failure info, like we do for matching and search requests, in it we have the FailureReason enum and a string with the exception message.