Handle incomplete transfer to anon

[p-j-smith]

Description

Fixes #506: Handle the cases where transfer from Raw to Anon fails or anonymisation fails, ensuring no identifiable data is stored on Anon

• Add an endpoint /import-from-raw to the Orthanc Anon plugin to retrieve a study from Orthanc Raw. Once retrieved:

  • the bytes for the study are downloaded
  • the study is then deleted so no PII is left in Orthanc Anon
  • the instances are anonymised
  • if anonymisation succeeds, the anonymised study is uploaded to Orthanc Anon and the export API is notified the study is ready for export

• Add a class PIXLAnonOrthanc to the imaging API. This is used to query Orthanc Raw (a remote modality for Orthanc Anon) and to call the import-from-raw endpoint

Type of change

Please delete options accordingly to the description.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

Suggested Checklist

• [ ] I have performed a self-review of my own code.
• [ ] I have made corresponding changes to the documentation.
• [ ] My changes generate no new warnings.
• [ ] I have commented my code, particularly in hard-to-understand areas.
• [ ] I have passed on my local host device. (see further details at the CONTRIBUTING document)
• [ ] Make sure your branch is up-to-date with main branch. See CONTRIBUTING for a general example to syncronise your branch with the main branch.
• [ ] I have requested PR review to UCLH-Foundtry/arc-dev
• [ ] I have adddressed and marked as resolved all the review comments in my PR.
• [ ] Finally, I have selected squash and merge

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 68.75000% with 15 lines in your changes missing coverage. Please review.

Project coverage is 87.15%. Comparing base (82ed836) to head (566b97b). Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pixl_imaging/src/pixl_imaging/_orthanc.py	59.25%	11 Missing :warning:
pixl_dcmd/src/pixl_dcmd/main.py	25.00%	3 Missing :warning:
pixl_imaging/src/pixl_imaging/_processing.py	90.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #512 +/- ## ========================================== - Coverage 87.32% 87.15% -0.18% ========================================== Files 76 76 Lines 3440 3464 +24 ========================================== + Hits 3004 3019 +15 - Misses 436 445 +9 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[p-j-smith]

The system tests are failing with:

Test failure traceback

```python 2024-10-14T14:39:28.132188683Z 2024-10-14 15:39:28.131 | WARNING | pixl:_anonymise_study_instances:334 - Failed to anonymize file: 2024-10-14T14:39:28.198773301Z E1014 15:39:28.198594 HTTP-41 PluginsManager.cpp:154] Error in the REST callback, traceback: 2024-10-14T14:39:28.198804299Z 2024-10-14T14:39:28.198812595Z Expecting value: line 1 column 1 (char 0) 2024-10-14T14:39:28.198819277Z 2024-10-14T14:39:28.198825409Z File "/etc/orthanc/pixl.py", line 271, in ImportStudyFromRaw 2024-10-14T14:39:28.198831991Z anonymised_instances_bytes, anonymised_study_uid = _anonymise_study_instances( 2024-10-14T14:39:28.198838263Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.198844374Z 2024-10-14T14:39:28.198850415Z File "/etc/orthanc/pixl.py", line 330, in _anonymise_study_instances 2024-10-14T14:39:28.198856778Z anonymised_instances_bytes.append(_anonymise_dicom_instance(dataset)) 2024-10-14T14:39:28.198862989Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.198869000Z 2024-10-14T14:39:28.198874671Z File "/etc/orthanc/pixl.py", line 227, in _anonymise_dicom_instance 2024-10-14T14:39:28.198880963Z anonymise_and_validate_dicom(dataset, config_path=None, synchronise_pixl_db=True) 2024-10-14T14:39:28.198887104Z 2024-10-14T14:39:28.198894047Z File "/.venv/lib64/python3.11/site-packages/pixl_dcmd/main.py", line 91, in anonymise_and_validate_dicom 2024-10-14T14:39:28.198901441Z dicom_validator = DicomValidator(edition="current") 2024-10-14T14:39:28.198907743Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.198913924Z 2024-10-14T14:39:28.198919575Z File "/.venv/lib64/python3.11/site-packages/pixl_dcmd/_dicom_helpers.py", line 54, in __init__ 2024-10-14T14:39:28.198925726Z self.dicom_info = EditionReader.load_dicom_info(json_path) 2024-10-14T14:39:28.198932459Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.198938500Z 2024-10-14T14:39:28.198944441Z File "/.venv/lib64/python3.11/site-packages/dicom_validator/spec_reader/edition_reader.py", line 173, in load_dicom_info 2024-10-14T14:39:28.198964188Z cls.load_info(json_path, cls.iod_info_json), 2024-10-14T14:39:28.198970540Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.198976712Z 2024-10-14T14:39:28.198982683Z File "/.venv/lib64/python3.11/site-packages/dicom_validator/spec_reader/edition_reader.py", line 167, in load_info 2024-10-14T14:39:28.198989045Z return json.load(info_file) 2024-10-14T14:39:28.198995126Z ^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.199001017Z 2024-10-14T14:39:28.199008150Z File "/usr/lib/python3.11/json/__init__.py", line 293, in load 2024-10-14T14:39:28.199014422Z return loads(fp.read(), 2024-10-14T14:39:28.199021[66](https://github.com/SAFEHR-data/PIXL/actions/runs/11329620982/job/31505577216?pr=512#step:14:67)6Z ^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.199028749Z 2024-10-14T14:39:28.199034169Z File "/usr/lib/python3.11/json/__init__.py", line 346, in loads 2024-10-14T14:39:28.199040181Z return _default_decoder.decode(s) 2024-10-14T14:39:28.199046412Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.199052573Z 2024-10-14T14:39:28.199058424Z File "/usr/lib/python3.11/json/decoder.py", line 337, in decode 2024-10-14T14:39:28.199064756Z obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 2024-10-14T14:39:28.1990[72](https://github.com/SAFEHR-data/PIXL/actions/runs/11329620982/job/31505577216?pr=512#step:14:73)431Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14T14:39:28.1990788[73](https://github.com/SAFEHR-data/PIXL/actions/runs/11329620982/job/31505577216?pr=512#step:14:74)Z 2024-10-14T14:39:28.199085615Z File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode 2024-10-14T14:39:28.199125220Z raise JSONDecodeError("Expecting value", s, err.value) from None 2024-10-14T14:39:28.199135849Z 2024-10-14T14:39:28.199190907Z E1014 15:39:28.198786 HTTP-41 PluginsErrorDictionary.cpp:101] Exception inside the plugin engine: Error encountered within the plugin engine ```

This also happens locally. However, if I spin up a local test instance and run the tests twice, they fail the first time and pass on the second run. Any ideas why this might be?

[p-j-smith]

thanks for the helpful reviews!