search

icgc-argo / argo-clinical

Clinical data submission for ARGO programs.
GNU Affero General Public License v3.0
2 stars 0 forks source link

Add Missing Entity Exceptions #1109

Closed joneubank closed 5 months ago

joneubank commented 5 months ago

Description of changes

Missing Entity Exceptions allow a submitting program that has lost contact with a donor before collecting treatment or followup information to use this donor's information with ARGO. Normally these clinical entities are required for the donor to be considered "core complete", and only data associated with a core complete donor is to be released. By granting a Missing Entity Exception for a donor we will be able to mark this donor as core complete despite the missing entities.

Connected Issues

This PR is a feature branch that will collect all the changes for the Issues connected the epic: https://github.com/icgc-argo/roadmap/issues/1084

Checklist

Type of Change

Checklist before requesting review:

joneubank commented 5 months ago

Attaching an image of the different entry points in the existing code base that lead to core-complete stat calculation. I've written this up because the code flow that leads to core completion calculation was difficult to follow at a glance. I will follow this comment with another that captures a new target plan for the code that also includes fetching missing entity exceptions.

The blue and green circles represent entry points to code execution. Green are HTTP API end points, and blue are GQL mutation requests. The total list of actions that trigger an update to core completion stats are:

  1. Submit clinical data (submission and registration updates)
  2. Patch core completion data - manual override mechanism
  3. Migrate schema version - Can result in different core clinical requirements

This process is complicated by the inclusion of code that lets an admin arbitrarily "override" an entity's completion state. This mechanism is available today but has not been well document and has not been used at all.

The main function that does the core completion calculation for a donor is calcDonorCoreEntityStats. This has two optional parameters recalcEvenIfComplete and recalcEvenIfOverridden.

recalcEvenIfComplete set to false means that the calculation will not be updated if it has already been done. When set to true this can result in a core-complete donor no longer being core-complete if the calculation logic requires more entities than the donor currently has.

recalcEvenIfOverridden will cause the completion state of a given entity to be checked even if it has an override. This is surprising since it can cause the existing override to be ignored, resulting in an inconsistent application of the override logic.

image

joneubank commented 5 months ago

Here is the a simplified view of the target for refactor.

We are going to remove the recalcEvenIf flags from the function that calculates core completion stats. The stats should calculate the same way every time.

We are also removing the override mechanism for core entities. This functionality is captured in the Missing Entity Exception rules. This will remove the patch core completion entry point and simplify the logic for calculating the core completion stats.

In place of this, we also need to provide the missing entity exception information to the calculation function. To fetch this information from the repo, we can create a new function called updateCompletionStatsForDonors that will fetch this exception information before calling calculateDonorCompletionStats to perform the actual calculation. This new function (updateCompletionStatsForDonors) can replace all the variants of update/recalculate/patch core completion stats that previously called calculateDonorCompletionStats.

image

Note:

Removing the core completion override functionality from the donor data model must be accompanied with a new migration to remove any reference to this property from the database.