Add functionality to allow for single/bulk delete or truncation of the DB

[f-odhiambo]

Need to delete dummy data from HAPI which includes dirty/dummy data while testing

• [ ] Add ability to either delete by resources

{
  "resourceType": "Bundle",
  "type": "transaction",
  "entry": [
    {
      "request": {
        "method": "DELETE",
        "url": "Organization/1"
      }
    },
    {
      "request": {
        "method": "DELETE",
        "url": "Organization/2"
      }
    }
  ]
}

• [ ] Mass Delete related Patient data via DELETE /Patient/123?_cascade=delete
• [ ] Delete resource + history via $expunge operation
• [ ] Delete all data on base

  
  POST [base]/$expunge
  Content-Type: application/fhir+json

{ "resourceType": "Parameters", "parameter": [ { "name": "expungeEverything", "valueBoolean": true } ] }

More here https://smilecdr.com/docs/fhir_repository/deleting_data.html

[lincmba]

I've conducted some research regarding the functionality:

1. Conditional Deletes: It seems that conditional deletes with a match URL like /Patient?_id=1,2 result in an error due to matching multiple resources. To overcome this, it might be necessary to delete resources one by one. ref: https://hl7.org/fhir/http.html#cdelete

2. Transactional Deletes: Transactional deletes, such as using a Bundle with DELETE methods, also encounter issues when a resource has references. Deleting references one by one might be necessary to successfully delete the target resource. ref: https://smilecdr.com/docs/fhir_repository/deleting_data.html#transactional-delete

3. Expunge Operation: The expunge operation is not supported as of now, and attempting to use it results in a parsing error. It might be worth checking if future updates of the tooling will include support for this operation. ref: https://smilecdr.com/docs/fhir_repository/deleting_data.html#instance-level-expunge

4. Cascading Deletes: Enabling allow_cascading_deletes and allow_multiple_delete at the JPA server settings seems to work for deleting multiple resources and all references at once. This means if a Patient of id 1 is referenced by another resource i.r Observation. When the patient is deleted, the referenced observation is deleted aswell. [DELETE] http://localhost:8888/fhir/Patient?_id=1,2&_cascade=delete ref: https://docs.oracle.com/en/industries/health-sciences/healthcare-data-repository/8.1.4/fhir-guide/deletes-and-referential-integrity.html#GUID-78537C7B-B9CC-466D-BFF4-8D75E785947C

I'll be implementing 4 as a solution to this

[Wambere]

@f-odhiambo @lincmba I'm probably missing some context here, but is there a reason we could not use this already existing cleaner ?

[f-odhiambo]

Option 4 does not exist on the cleaner , so should we add this ad an enhancement there ?

[pld]

since this already done in the PR, let's make sure the PR matches the cleaner functionality and remove the cleaner, I think we need to add expunge to the efsity version to match.

@lincmba you can update the PR to delete the cleaner, after adding the expunge functionality

[Wambere]

@pld after some discussions with @ndegwamartin and @lincmba we think that it makes more sense to update the current cleaner and keep using it instead of deleting it, since it has the following functionality that does not exist here yet

1. Takes in any parameter and value combination used to filter/find the resources you want to delete, while this takes in only IDs
2. Able to expunge records after deleting
3. Transaction requests - deleting in batches, while this deletes one at a time

We would need to add the cascade delete functionality to the cleaner though, creating a separate issue for that

[pld]

Ah ok, so then we're remove the cleaning functionality from esfity? Or maybe better, merge that PR, so it's there and working, but at a doc string pointing to the cleaner tool?

[pld]

Nevermind, just saw this comment, https://github.com/onaio/fhir-tooling/issues/101#issuecomment-1893980465

[Wambere]

To be completed in https://github.com/onaio/fhir-tooling/issues/106