Deleting concept sets/cohort definitions and impact on analyses

[anthonysena]

Discussion points for v3.0

• Should we consider only performing "soft deletes" in the system? I'd favor this so that we can avoid problems like those listed above.

• Aging out deleted entities: in the case of cohort definitions, deleting the definition triggers a clean up routine to clear out the data for that cohort in the results schema. If we soft delete an entity, we may want to have a cleanup run but only after certain period of time has passed. In addition, we may want to physically delete the entity definition but this then creates a more complex problem where we need to understand all of the potential relationships and entity ages. Perhaps we need to define a "reference" entity type vs an "analysis" entity that can utilize a reference entity? An example of a reference entity would be a cohort definition while an analysis entity could be a characterization.

• Administration of deleted entities: will a user have the ability to restore an entity? Do we need further permissions to enable an administrator to restore an entity?

• Understanding dependencies: if we adopt the idea of a "reference" entity, we'll need to have an endpoint to see what "analysis" entities use it. This would include the status (deleted/active) of the analysis.

[pavgra]

@anthonysena, how have you removed a cohort used in e.g. Pathways analysis? Here is what I get while trying to do that (using rc-2.7.4 branch):

[pavgra]

And here is what I have when I attempt to remove the cohort directly from DB:

[pavgra]

Public Atlas:

[anthonysena]

@pavgra - the items above are related to investigating #1329 which I've closed out. Let's use this issue to discuss approaches for deletion of dependencies in general in Atlas/WebAPI in the v3.0 stack.