Automatically (Soft-)Delete Samples & Pathogen Tests with Negative Test Results for COVID-19 [3]

markusmann-vg commented 3 weeks ago

Problem Description

As a system administrator for the Luxembourg health department, I want to automatically soft-delete samples with negative test results for COVID-19, so that the system can be kept clean of irrelevant test data and maintain database efficiency.

Proposed Change

[ ] query to select samples that meet the conditions.
[ ] The nightly job is updated to move these samples into the soft-delete state automatically.
[ ] The system properly reflects these samples as soft-deleted, ensuring they are not visible in normal workflows but remain available for auditing if needed.

Added Value/Benefit

Database Optimization: Automatically removing irrelevant data such as negative test results helps keep the database leaner and more efficient. This reduces the amount of unnecessary data stored, improving overall system performance, including query response times and storage management.

Data Hygiene and Quality: Keeping the database free of outdated or irrelevant negative test results helps maintain data cleanliness, reducing the risk of errors and inconsistencies when working with the data. This is crucial for public health systems where the quality of information is vital.

Automated Data Management: Automating the soft-deletion process reduces the need for manual intervention, saving time for system administrators and health department staff. This also minimizes the risk of human error, ensuring that irrelevant data is consistently and accurately managed.

Acceptance Criteria

Nightly Job Extension:

[ ] The existing nightly job, which automatically deletes entities marked for deletion, must be extended to include COVID-19 samples with negative results that meet the specified criteria. Criteria for Soft Deletion:

The query must capture samples that meet the following conditions: => Search negative covid-19 pathogen-tests => delete pathogen-tests

delete negative samples where all covid-19 pathogen tests were deleted
=> Search Samples without a covid-19 pathogen test and no other test => delete samples

Soft Delete Implementation:

[ ] Samples meeting the criteria must be soft-deleted (i.e., marked as deleted but not fully removed from the system). Pending Information:
[ ] The number of days after which negative samples should be deleted is not yet available and should be integrated once received. => Assumption 90 days (needs to be higher than a normal test duration)

Implementation Details

No response

Mockups

No response

Additional Information

Out of scope Event-Participant Disease Management (UI Warning if the automatic deletion is below x days)

No response

leventegal-she commented 1 week ago

To be added to the release notes:

New configuration property: negativeCovidTestsMaxAgeDays: Number of days after which negative CORONAVIRUS pathogen tests and their samples are soft deleted default: not set, meaning no delete possible values: any integer representing the number of days

richardbartha commented 2 days ago

Tested on localhost version 1.99.0 - SNAPSHOT (https://github.com/SORMAS-Foundation/SORMAS-Project/commits/0c557021cc987a83391e4d054f44f55ede18e358)

SORMAS-Foundation / SORMAS-Project