search

aces / Loris-MRI

The set of scripts that preprocess and insert MRI data into the database.
10 stars 51 forks source link

Unify LORIS-MRI logging #1191

Open maximemulder opened 2 months ago

maximemulder commented 2 months ago

Description of the PR

This PR is a rewrite of the logging and I/O modules of LORIS-MRI.

Problems of the current architecture

There are several problems with the current logging and I/O architecture of LORIS-MRI:

  1. It is fragmented. The core logging class is the Log class. The BasePipeline and ImagingIO classes build on top of Log provide better abstractions, with some amount of duplication between those two classes (some scripts simply use print but they are not the subject of this PR).
  2. The BasePipeline class is too coupled with the DCM2BIDS pipeline to be generalized.
  3. The BasePipeline and ImagingIO classes handle both logging and file system operations, which are two different responsibilities.
  4. The logging abstractions are verbose, notably because all logging functions require two keyword arguments is_error = 'Y' | 'N' and is_verbose = 'Y' | 'N'.
  5. The logging functions sometimes uses stdout to print errors, whereas they should use stderr.
  6. The code does not integrate the latest LORIS-MRI architecture improvements, namely static typing and the SQLAlchemy integration.

Description of the new architecture

This PR adds the following:

  1. An lib.env.Env dataclass that stores generic information about a running script.
  2. The lib.logging module that handles log operations.
  3. The lib.file_system module that handles file system operations.
  4. The make_env function that allows to build an Env from a LorigGetOpt object (note that the environment does not directly depend on LorisGetOpt, it could also be built manually).

Those abstractions aim to replace the Log and ImagingIO classes, as well as the BasePipeline logging methods. They are script-agnostic, remove the existing duplication, are statically typed, use the SQLAlchemy abstractions, and provide more convenient interfaces IMO. They also use simple functions instead of classes, which I think is better for simplicity, reusability and testability.

Backwards compatibility

This PR is backwards compatible as the old abstractions are not replaced by this PR. They are however annotated as depcreated to let some time for external scripts to move to the old abstractions. All the uses of the old abstractions in LORIS-MRI are replaced by the new abstractions in this PR, so the deprecated files remain only for external scripts compatibility.

Ultimately (once the migration is finished), I would like the new logging module to take the lib.log namespace though (the same way I would like the new database module to use the lib.database module).

Final notes

I am not an expert in logging, but I am not a fan of how logging is done in LORIS-MRI (printing AND writing to a log file that does not differentiate stdout and stderr AND inserting into the database on every print). I much prefer how the LORIS process manager does it, which is redirecting stdout and stderr to some files, and saving these files paths in the database. Nevertheless, the current behaviour is conserved in this PR as it is only an architectural change, and I believe this PR is an improvement no matter how LORIS-MRI logging evolves in the future.

maximemulder commented 1 month ago

Tested using batch_uploads_imageuploader.pl (with dcm2niix, so that it runs the Python pipeline) and it works fine (although this script produces a very ugly output in general, but that is not the subject of this PR) !

If you want to inspect the diff yourself, here is what I got:

Main:

(loris-mri-python) lorisadmin@mmulder-dev:/data/test_upload$ echo "DCC422_475906_V2.tar.gz N DCC422_475906_V2" | batch_uploads_imageuploader.pl -profile prod
Running now the following command: /opt/bic/bin/mri/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 140 DCC422_475906_V2.tar.gz -verbose

 find \/tmp\/ImagingUpload\-10\-5\-Wl4f_6 -name '__MACOSX' -exec rm -rf {} +  

dicomTar.pl \/tmp\/ImagingUpload\-10\-5\-Wl4f_6 \/data\/raisinbread\/tarchive -database -profile prod -verbose 
 Source: /tmp/ImagingUpload-10-00-abcdef
Target: /data/raisinbread/tarchive

Testing for database connectivity.
Database is available.

You will archive the dir        : ImagingUpload-10-00-abcdef

You are creating a tar with the following command:

tar -cf /data/raisinbread/tarchive/ImagingUpload-10-00-abcdef.tar ImagingUpload-10-00-abcdef

getting md5sums and gzipping!!

* Taken from dir                   :    /tmp/ImagingUpload-10-00-abcdef
* Archive target location          :    /data/raisinbread/tarchive/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar
* Name of creating host            :    132.206.37.11
* Name of host OS                  :    Linux
* Created by user                  :    lorisadmin
* Archived on                      :    2024-10-15 10:06:13
* dicomSummary version             :    1
* dicomTar version                 :    1
* md5sum for DICOM tarball         :    baab0f21d6404d75dd3d3e0bc9d84a5e  ImagingUpload-10-00-abcdef.tar
* md5sum for DICOM tarball gzipped :    ea1f021c7e67cc1c45fd8ae32d023c2b  ImagingUpload-10-00-abcdef.tar.gz
* md5sum for complete archive      :    02788d3b64b721de12e7c61c8eb97b41  DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar

Adding archive info into database

Removing temporary files from target location

Done adding archive info into database

\/opt\/bic\/bin\/mri/python/scripts/run_dicom_archive_loader.py -p database_config.py -u 140 -v 
 ==> Successfully connected to database

==> Found Center Name: DCC, Center ID: 1

==> Found Scanner ID: 4

==> run_dicom_archive_validation.py successfully executed for UploadID 140 and ArchiveLocation /data/raisinbread/tarchive/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar

==> b'Chris Rorden\'s dcm2niiX version v1.0.20211006  (JP2:OpenJPEG) GCC11.2.0 x86-64 (64-bit Linux)\nFound 352 DICOM file(s)\nConvert 160 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4 (256x256x160x1)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00003 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00003.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00001 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00001.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00002 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00002.nii"\nConvert 160 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3 (224x256x160x1)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00002 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00002.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00003 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00003.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00001 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00001.nii"\nConvert 26 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5 (96x96x81x26)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii"\nConversion required 18.568907 seconds (1.656919 for core code).\n'

==> Number of NIfTI files that will be considered for insertion into the database: 4

==> run_nifti_insertion.py successfully executed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3.nii.gz

==> run_nifti_insertion.py successfully executed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4.nii.gz

b'==> Successfully connected to database\n\n==> Found Center Name: DCC, Center ID: 1\n\n==> Found Scanner ID: 4\n\n==> Session ID for the file to insert is 2143\n\n==> Acquisition protocol ID for the file to insert is 66\n\n==> Creating directory /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n\n\n[ERROR   ] Scan type dwi25 does not have BIDS tables set up.\n\n'
==> run_nifti_insertion.py failed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz.
b'==> Successfully connected to database\n\n==> Found Center Name: DCC, Center ID: 1\n\n==> Found Scanner ID: 4\n\n==> Session ID for the file to insert is 2143\n\n==> Acquisition protocol ID for the file to insert is 66\n\n==> Creating directory /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\n\n==> Moving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n\n==> Moving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n\n\n[ERROR   ] Scan type dwi25 does not have BIDS tables set up.\n\n'

==> 
        Finished processing UploadID 140!
        - DICOM archive info: 80 => /data/raisinbread/tarchive/2018/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar
        - 2 files were inserted into the files table: assembly_bids/sub-475906/ses-V2/anat/sub-475906_ses-V2_run-1_T1w.nii.gz, assembly_bids/sub-475906/ses-V2/anat/sub-475906_ses-V2_run-1_T2w.nii.gz
        - 1 files did not match any protocol: trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz
        - 0 files were exclusionary violations: 0
        - Log of process in /data/raisinbread/logs/run_dicom_archive_loader/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh.log

find /data/raisinbread/tarchive/ -name *ImagingUpload-10-00-abcdef* 

rm /data/test_upload/DCC422_475906_V2.tar.gz

This PR:

(loris-mri-python) lorisadmin@mmulder-dev:/data/test_upload$ echo "DCC422_475906_V2.tar.gz N DCC422_475906_V2" | batch_uploads_imageuploader.pl -profile prod
Running now the following command: /opt/bic/bin/mri/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 141 DCC422_475906_V2.tar.gz -verbose

 find \/tmp\/ImagingUpload\-10\-15\-d5v5D4 -name '__MACOSX' -exec rm -rf {} +  

dicomTar.pl \/tmp\/ImagingUpload\-10\-15\-d5v5D4 \/data\/raisinbread\/tarchive -database -profile prod -verbose 
 Source: /tmp/ImagingUpload-10-00-abcdef
Target: /data/raisinbread/tarchive

Testing for database connectivity.
Database is available.

You will archive the dir        : ImagingUpload-10-00-abcdef

You are creating a tar with the following command:

tar -cf /data/raisinbread/tarchive/ImagingUpload-10-00-abcdef.tar ImagingUpload-10-00-abcdef

getting md5sums and gzipping!!

* Taken from dir                   :    /tmp/ImagingUpload-10-00-abcdef
* Archive target location          :    /data/raisinbread/tarchive/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar
* Name of creating host            :    132.206.37.11
* Name of host OS                  :    Linux
* Created by user                  :    lorisadmin
* Archived on                      :    2024-10-15 10:15:59
* dicomSummary version             :    1
* dicomTar version                 :    1
* md5sum for DICOM tarball         :    011704f1fb6539d5d6c7c097903a26ad  ImagingUpload-10-00-abcdef.tar
* md5sum for DICOM tarball gzipped :    e18ecf0ac53b8f9a440e2baf59979a1e  ImagingUpload-10-00-abcdef.tar.gz
* md5sum for complete archive      :    a0a16172cc1c4d3c9c2dd25bd7d9f856  DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar

Adding archive info into database

Removing temporary files from target location

Done adding archive info into database

\/opt\/bic\/bin\/mri/python/scripts/run_dicom_archive_loader.py -p database_config.py -u 141 -v 
 Connecting to the database using the following configuration:
  Host: mariadb-dev0.acelab.ca
  Port: 3306
  Username: mmulder_dev_user
  Database: mmulder_dev
  (password hidden)
Successfully connected to the database
Found Center Name: DCC, Center ID: 1
Found Scanner ID: 4
run_dicom_archive_validation.py successfully executed for UploadID 141 and ArchiveLocation /data/raisinbread/tarchive/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar
b'Chris Rorden\'s dcm2niiX version v1.0.20211006  (JP2:OpenJPEG) GCC11.2.0 x86-64 (64-bit Linux)\nFound 352 DICOM file(s)\nConvert 160 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4 (256x256x160x1)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00003 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00003.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00001 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00001.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00002 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_1_i00002.nii"\nConvert 160 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3 (224x256x160x1)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00002 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00002.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00003 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00003.nii"\nConvert 1 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00001 (256x256x1x1)\nWarning: Check that 2D images are not mirrored.\nCompress: "/usr/bin/pigz" -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_localizer_20180202143610_2_i00001.nii"\nConvert 26 DICOM as /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5 (96x96x81x26)\nCompress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii"\nConversion required 12.096276 seconds (0.801164 for core code).\n'
Number of NIfTI files that will be considered for insertion into the database: 4
run_nifti_insertion.py successfully executed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t1_mpr_1mm_p2_pos50_20180202143610_3.nii.gz
run_nifti_insertion.py successfully executed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_t2_spc_1mm_p2_20180202143610_4.nii.gz
b'ERROR: Scan type dwi25 does not have BIDS tables set up.\nConnecting to the database using the following configuration:\n  Host: mariadb-dev0.acelab.ca\n  Port: 3306\n  Username: mmulder_dev_user\n  Database: mmulder_dev\n  (password hidden)\nSuccessfully connected to the database\nFound Center Name: DCC, Center ID: 1\nFound Scanner ID: 4\nSession ID for the file to insert is 2143\nAcquisition protocol ID for the file to insert is 66\nCreating directory /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n'
run_nifti_insertion.py failed for file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz.
b'ERROR: Scan type dwi25 does not have BIDS tables set up.\nConnecting to the database using the following configuration:\n  Host: mariadb-dev0.acelab.ca\n  Port: 3306\n  Username: mmulder_dev_user\n  Database: mmulder_dev\n  (password hidden)\nSuccessfully connected to the database\nFound Center Name: DCC, Center ID: 1\nFound Scanner ID: 4\nSession ID for the file to insert is 2143\nAcquisition protocol ID for the file to insert is 66\nCreating directory /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.json\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bval\nMoving file /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\nMoving /tmp/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh/nifti_files/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec to /data/raisinbread/trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.bvec\n'

        Finished processing UploadID 141!
        - DICOM archive info: 81 => /data/raisinbread/tarchive/2018/DCM_2018-02-02_ImagingUpload-10-00-abcdef.tar
        - 2 files were inserted into the files table: assembly_bids/sub-475906/ses-V2/anat/sub-475906_ses-V2_run-1_T1w.nii.gz, assembly_bids/sub-475906/ses-V2/anat/sub-475906_ses-V2_run-1_T2w.nii.gz
        - 1 files did not match any protocol: trashbin/run_nifti_insertion_2024-10-15_10h00m00s_abcdefgh/ImagingUpload-10-00-abcdef_ep2d_dif_25dir_NO_ANGLE_20180202143610_5.nii.gz
        - 0 files were exclusionary violations: 0
        - Log of process in /data/raisinbread/logs/run_dicom_archive_loader/run_dicom_archive_loader_2024-10-15_10h00m00s_abcdefgh.log

find /data/raisinbread/tarchive/ -name *ImagingUpload-10-00-abcdef* 

rm /data/test_upload/DCC422_475906_V2.tar.gz

I used the following scripts to uniformize the timestamps across the diffs:

#!/bin/sh

cp main.txt main_copy.txt
cp pr.txt pr_copy.txt
sed -i -e 's/10-5-Wl4f_6/10-00-abcdef/g' main_copy.txt
sed -i -e 's/10-15-d5v5D4/10-00-abcdef/g' pr_copy.txt

sed -i -e 's/10h06m53s_hv0qp2_x/10h00m00s_abcdefgh/g' main_copy.txt
sed -i -e 's/10h08m15s_isxuj3lf/10h00m00s_abcdefgh/g' main_copy.txt

sed -i -e 's/10h16m19s_szr6cbl0/10h00m00s_abcdefgh/g' pr_copy.txt
sed -i -e 's/10h17m12s_yyfmqxed/10h00m00s_abcdefgh/g' pr_copy.txt

The changes that remain are mostly:

  1. IDs that vary across runs.
  2. Removal of ==> in output (which is good IMO).
  3. Order changes because the order in which things run is not deterministic (and so changes across runs).

I also checked that the log file exists in the LORIS log directory and that everything was logged in the database.

maximemulder commented 1 month ago

Tested it with extract_eeg_bids_archive.py.

Main:

(loris-mri-python) lorisadmin@mmulder-dev:~$ extract_eeg_bids_archive.py --profile database_config.py --upload_id 6 --verbose
==> Moving /tmp/extract_eeg_bids_archive_2024-10-23_15h56m14s_pylwgb2z/EEG_DIR_2024-10-23_15h56m14s_cdpcsgc6/OTT166_300166_V1/sub-OTT166/ses-V1/eeg to /data/raisinbread/assembly_bids/sub-OTT166/ses-V1/eeg

This PR:

(loris-mri-python) lorisadmin@mmulder-dev:~$ extract_eeg_bids_archive.py --profile database_config.py --upload_id 6 --verbose
Connecting to the database using the following configuration:
  Host: mariadb-dev0.acelab.ca
  Port: 3306
  Database: mmulder_dev
  Username: mmulder_dev_user
  (password hidden)
Successfully connected to the database
Moving /tmp/extract_eeg_bids_archive_2024-10-23_15h48m29s_7cgdq0wk/EEG_DIR_2024-10-23_15h48m29s__sf6kr14/OTT166_300166_V1/sub-OTT166/ses-V1/eeg to /data/raisinbread/assembly_bids/sub-OTT166/ses-V1/eeg