Closed daria-sorokina-da closed 2 months ago
In Anthony Nolan DEV environment:
It finished successfully, seeing results in [MatchingAlgorithmPersistent].[DataRefreshHistory] table: | Id | Database | WasSuccessful | RefreshRequestedUtc |
---|---|---|---|---|
113 | DatabaseA | 1 | 2024-06-26 10:09:58.8216793 |
Expected: The value in [LastUpdateDateTime] column is filled for all donors, and it is the same as in the persistent [Donors].[Donors] database [LastUpdated] column for that donor.
Actual:
The value in [LastUpdateDateTime] column is "0001-01-01 00:00:00.0000000 +00:00" for all donors.
See:
Additional Notes:
Records produced by subsequent Donor Updates (outside of Data Refresh process) have the same issue:
$\textcolor{red}{\textsf{Testing failed}}$
It finished successfully, seeing results in [MatchingAlgorithmPersistent].[DataRefreshHistory] table: | Id | Database | WasSuccessful | RefreshRequestedUtc |
---|---|---|---|---|
114 | DatabaseB | 1 | 2024-07-01 14:46:27.9961012 |
Expected: There are entities in the [dbo].[DonorManagementLogs] table. The table is empty.
Actual: There are no entities in the [dbo].[DonorManagementLogs] table. The entities are present in the [dbo].[Donors] table though.
$\textcolor{red}{\textsf{Testing failed}}$
This happened because when a nomenclature version does not changes since the last successful Data Refresh, then the Data Refresh is run with ShouldMarkAllDonorsAsUpdated = false setting, and the [dbo].[DonorManagementLogs] table does NOT get populated. This is incorrect behavior, because in this case Repeat Searches will not be returning donors that changed between the last search, and the Data Refresh (several days may pass between them).
Always set ShouldMarkAllDonorsAsUpdated = true for all data refreshes as part of 2.2.0. In the future (Atlas 3+), ShouldMarkAllDonorsAsUpdated flag should be removed from the Data Refresh parameters and from the [MatchingAlgorithmPersistent].[DataRefreshHistory] database.
It finished successfully, seeing results in [MatchingAlgorithmPersistent].[DataRefreshHistory] table: | Id | Database | WasSuccessful | RefreshRequestedUtc |
---|---|---|---|---|
115 | DatabaseA | 1 | 2024-07-02 11:16:05.1224643 |
Result: There are entities in the [dbo].[DonorManagementLogs] table. The donors have LastUpdateDateTime set from persistent database Donors.Donors table.✔️
SELECT TOP(1000) dml.*, d.ExternalDonorCode
FROM dbo.DonorManagementLogs dml
JOIN dbo.Donors d on d.DonorId = dml.DonorId
ORDER BY dml.Id DESC
Expected: An entry about updated donor is added with LastUpdatedDateTime - now.
Actual: An entry about updated donor is added with LastUpdatedDateTime "0001-01-01 00:00:00.0000000 +00:00" ❌
$\textcolor{red}{\textsf{Testing failed}}$
Context
Currently, during Data Refresh, the LastUpdateDateTime column in the DonorManagementLogs in transient (A/B) databases is set to the date and time of Date Refresh processing.
Example: in the WMDA Live environment the last Data Refresh happened on 2024-04-26, and in the DonorManagementLogs we see lots of records with LastUpdateDateTime 2024-04-26, see:![image](https://github.com/Anthony-Nolan/Atlas/assets/84312905/571799ac-2ccb-4073-94dd-2c11beeee80f)
Here "-1" in the SequenceNumberOfLastUpdate indicates that the Data Refresh process created this record.
Issue description
This approach causes issues during Repeat Searches run after the data refresh. That is because all the donors added during Data Refresh now have LastUpdateDateTime after the last cut-off date of the search. So basically Repeat Searches perform the search from scratch (return all the same donors as the original search, as if they changed).
This causes performance issues on Atlas side and increases the costs in Azure, and causes performance issues on the client side as well, as the client does not expect the Repeat Searches to return lots of results.
Proposed fix
We need to start filling the LastUpdateDateTime with the date when the donor was last updated, instead of the Data Refresh date. We should take the date from the persistent database Donors.Donors table - LastUpdated.
Solution details
Atlas.DonorImport.Data.Repositories
DonorReadRepository.StreamAllDonors
update the returned model - add LastUpdated date to it.Atlas.MatchingAlgorithm.Services.DataRefresh.DonorImport
DonorImporter.InsertDonorBatch
- instead ofUpdateDateTime = batchFetchTime
we should now haveUpdateDateTime = d.LastUpdated