New dicom archive

[MaximeMulder]

Description

This PR is a rewrite (and cleaning/modernization) of dicom-archive/dicomTar.pl, dicom-archive/dicomSummary.pl and dicom-archive/DICOM/DCMSUM.pm in Python, which intends to replace them in the long term. This PR does not replace them for now as we should do more testing and integrate these scripts with the rest of the pipeline before doing so, likely in the next LORIS version imo.

Presentation

This PR adds three Python scripts:

• dicom_summary.py, which is the port of dicomSummary.pl and generates a textual summary of a DICOM directory (surprise !).
• dicom_archive.py, which is the port of dicomTar.pl, creates a DICOM directory archive for and (optionally) inserts or updates its information in the LORIS database.

To support these scripts, the python/lib/dicom/ directory was added, which contains various modules used by the aforementioned scripts.

• The summary_*.py files contain code for the DICOM summary.
• The dicom_log.py file contains code for the DICOM archiving log.
• The text_*.py files contains text utilities code used by the two previous points.
• The dicom_database.py file contains the code used to insert a DICOM directory's information in the database.

Breaking changes

I removed the following options from dicomTar.pl / dicom_archive.py, as they did not seem to be used by any other script:

• -centerName, which fills the tarchive.neurodbCenterName field in the database, but seems unused in LORIS and LORIS-MRI.
• -mri_upload_update, which launchs the updateMRI_Upload.pl script after dicomTar.pl and also seems unused (but less confident about that one).
• I also think splitting -clobber into --db-update (to update instead of inserting in the database) and --owerwrite (to replace files on the disk if they exist) might be a good idea.
• I also added the --year option, which adds the created DICOM archive in a dedicated year subdirectory (according to the scan date), which is convenient for me at the BIC.

I did not port the tools/addSeriesAndFileRecords.pl script as it seems to not be used in LORIS / LORIS-MRI, and should maybe be deleted in the long term. It is however very easy to re-code with the new architecture (it is currently already a simple script).

PLEASE TELL ME IF ANYONE HAS A PROBLEM WITH THESE CHANGES. Other suggestions are also welcome.

Remaining work

Before merging:

• Add a markdown documentation for the scripts.
• Do more testing.
• More exhaustive error handling.

Longer term work:

• Integrate the new scripts with the existing pipelines and remove the old ones.
• Clean up the tarchive table (remove the neurodbCenterName / LastUpdated fields ? make CenterName nullable ?).

[MaximeMulder]

Okay, I think the PR is finished, I am happy with the code and tested it on a few DICOMs on my side.

The tests are failing because I am using a few match statements, which were added in Python 3.10. So I'd like to bump the Python minimum supported version to 3.10, if not, I can remove them (but obviously prefer the newer and shinier version).

We should also test it on DICOMs from other projects. Then, when this PR is merged, I can make another PR to adopt it in the different superscripts, although a problem will be that this script requires the Python profile file, while the scripts that call dicomTar.pl use the Perl one (so both will probably need to be passed as arguments).

I will note a few things on the code:

• I did not use our custom argument parser because there is now argparse in the standard Python library, which handles all the cases we want. We probably want to transition to that long term.
• I also wrote a few database helpers for my code, but I think in the long term we should transition to a database and query abstraction library, like sqlalchemy which I've heard about.

[MaximeMulder]

Blocked on Python 3.10, as well as #1141 and #1142 to see if this PR should be updated with new libraries.

NOTE for myself: Bring a fix from the new-mri-upload branch before merging (I don't remember which one but it should be easy) and do not initiate the database connection if the profile is not supplied.