Create a git-annex repo for data from `basel/4031_MRI_T2_MRAGE_24patients_CGM`

plbenveniste commented 4 months ago

The data is stored in duke/mri/basel/4031_MRI_T2_MRAGE_24patients_CGM and the segmentations are stored in duke/mri/basel/4031_Segmentations_T2_MPRAGE_24patients_CGM The steps are the following:

[x] A git-annex repo should be created :
[x] The data should be Bidsfied
[x] The data should be pushed to the repo
[x] A PR should be opened
[x] The PR should be merged.

@jcohenadad What name should be used for the git-annex repo ? ms-basel-mprage-4031 ?

jcohenadad commented 4 months ago

What name should be used for the git-annex repo ? ms-basel-mprage-4031 ?

given that the dataset equally has MPRAGE and TSE, I would not call it 'mprage' only. Also, '4031' is not super-specific (and we don't know what it means). Maybe the year it was shared...?

also not sure what CGM means (could be part of the questions for Haris)

plbenveniste commented 2 months ago

I looked for more information regarding the data and could only find the following:

the scans were done in 2017 and 2018
the dataset contains T2 TSe and T1 MPRAGE images
the images are in the DCM format while the segmentations are stored in .nii.gz

I don't know what 4031 stands for. I don't know what CGM stands for as well. Maybe "Cortical Gray Matter"?

From this, I can't really figure out a name for this data.

What do you think about simply ms-basel or ms-basel-2017-2018 (for the moment when the images were acquired)? (To have something different from this data #20 which could be ms-basel-2017-2019)

jcohenadad commented 2 months ago

how about ms-basel-2018 (end date of acquisition) and the other could end with 2019?

plbenveniste commented 2 months ago

Works for me !

@mguaypaq Could you create the corresponding git-annex repo please?

plbenveniste commented 2 months ago

As I am running a script to convert the files from dcm to nii, I encountered the following error for two files (5934_CGM_20171214_0003_t1_mprage_sag_iPat2_64CH, 6622_CGM_20180208_0004_t1_mprage_sag_iPat2_64CH). The error is the following:

Chris Rorden's dcm2niiX version v1.0.20240202  GCC12.3.0 x86-64 (64-bit Linux)
Found 119 DICOM file(s)
Warning: Instance Number (0020,0013) order is not spatial.
Warning: Interslice distance varies in this volume (incompatible with NIfTI format).
Warning: Missing images? Expected 119 images, but instance number (0020,0013) ranges from 120 to 1
Convert 119 DICOM as /home/GRAMES.POLYMTL.CA/p119007/create_basel_ms_2017_2018/ms-basel-2017-2018/sub-5934/ses-20171214/anat/sub-5934_ses-20171214_acq-sagMPRAGE_T1w (256x256x119x1)
Warning: Unable to rotate 3D volume: slices not equidistant: 1 != 2
Conversion required 2.871353 seconds (2.871186 for core code).

What I found from looking into the subject is that:

for 5934 one of of the 120 files is empty (zero bites): The file is corrupted (missing one slice)
for 6622 it contains only 69 files instead of 120: the file is corrupted (doesn't contain the spinal cord)
the files can still be opened with ITK Snap
dcm2bids_helper is able to convert it and it uses the same parameters as I do:

Running: dcm2niix -b y -ba y -z y -f %3s_%f_%p_%t -o /home/GRAMES.POLYMTL.CA/p119007/create_basel_ms_2017_2018/test /home/GRAMES.POLYMTL.CA/p119007/create_basel_ms_2017_2018/4031_MRI_T2_MRAGE_24patients_CGM/5934_CGM_20171214_0003_t1_mprage_sag_iPat2_64CH/

My opinion is to discard these two files and not convert them to BIDS.

plbenveniste commented 2 months ago

Something surprising I found is that, the segmentations of these two problematic files exist. They contain lesions and are not corrupted. They seem to both contain 120 slices. Maybe the data was corrupted when it was sent to us. What should we do ?

In the mean time, I am discarding the segmentations which don't have an associated image.

plbenveniste commented 2 months ago

The images and lesion segmentations were bidsified using the following command:

python code/convert_dcm_2_bids.py --i ~/create_basel_ms_2018/4031_MRI_T2_MRAGE_24patients_CGM/ --o ~/create_basel_ms_2018/ms-basel-2018/ --s ~/create_basel_ms_2018/4031_Segmentations_T2_MPRAGE_24patients_CGM/

plbenveniste commented 2 months ago

@jcohenadad By whom was this data share (for the README.md) ? And do we know who did the manual segmentation (to create the associated json sidecar) ?

jcohenadad commented 2 months ago

I just noticed that the ZIP code of the center is '4031'. Coincidence?

University Hospital Basel Department of Neurology Petersgraben 4 CH-4031 Basel

jcohenadad commented 2 months ago

By whom was this data share (for the README.md) ?

Haris Tsagkas, Cristina Granziera

And do we know who did the manual segmentation (to create the associated json sidecar) ?

Haris Tsagkas

plbenveniste commented 2 months ago

All done on my side ! For the lesion segmentation, I used the date of last modification of the file (which is 2022-10-13 for every image). Waiting for the repo to be created so that the data can be pushed and reviewed before being merged.

plbenveniste commented 2 months ago

I found some issues when reviewing the segmentations (based on some observations I made on #20 ).

There was an issue with a file name (fixed).
The files need to be converted to uint8 (fixed).
The segmentations are not in the same orientation and resolution as the images. I am therefore resampling to the resolution of the images (on it)

plbenveniste commented 2 months ago

I am currently facing an issue: When I resample using the following line:

nx, ny, nz, nt, px, py, pz, pt = img.dim
resampling = f'{str(px)}x{str(py)}x{str(pz)}'
os.system(f"sct_resample -i {seg_file} -x linear -mm {resampling} -o {seg_file}")

Which is:

sct_resample -i /home/GRAMES.POLYMTL.CA/p119007/create_basel_ms_2018/ms-basel-2018/derivatives/labels/sub-2871/ses-20171222/anat/sub-2871_ses-20171222_acq-sagTse_T2w_lesion-manual.nii.gz -x linear -mm 0.5729167x0.5729167x3.0 -o /home/GRAMES.POLYMTL.CA/p119007/create_basel_ms_2018/ms-basel-2018/derivatives/labels/sub-2871/ses-20171222/anat/sub-2871_ses-20171222_acq-sagTse_T2w_lesion-manual.nii.gz

After resampling, I open the segmentation again to compare the dimensions and I get the following: (384, 384, 15, 1, 0.5729167, 0.5729167, 3.0, 1) (447, 447, 40, 1, 0.57270694, 0.57270694, 2.999996, 1)

The difference between the first three dimension is because the segmentation mask needs to be cropped so that it is the same shape as the image. I am thinking to approach this with the following command line (is that correct?):

sct_crop_image -i seg.nii.gz -ref img.nii.gz

However, there is still a problem with the voxel dimension after (value 4 to 6). I am wondering if this linked to an approximation (to the 3rd decimal) done somewhere in SCT. Any ideas on this? What should I do?

jcohenadad commented 2 months ago

I would not resample within the src image (seg) but rather apply a transformation based on the qform of the src and the dest image (MRI image). That way you don’t need to deal with rounding issues. Eg of syntax: sct_register_multimodal -i XXX -d XXX -identity 1

plbenveniste commented 2 months ago

This method works great! I think It should be the go-to method when correcting segmentations masks, as it deals with a lot of issues at the same time.

The script used does the following now:

moves the images in the bids format folder
formats the segmentation files and creates the json sidecar files.
writes the participants.tsv and participants.json files.
writes the dataset_description.json file.
reorients and resamples the segmentation files.
binarizes the segmentation files.
updates the json sidecar files of the segmentation files.
checks that all files are stored in the uint8 format.

It was ran using:

python code/convert_dcm_2_bids.py --i ~/create_basel_ms_2018/4031_MRI_T2_MRAGE_24patients_CGM/ --o ~/create_basel_ms_2018/ms-basel-2018/ --s ~/create_basel_ms_2018/4031_Segmentations_T2_MPRAGE_24patients_CGM/

Ready to be pushed when repo is ready

mguaypaq commented 2 months ago

Sorry for the delay, the repository is now created and @plbenveniste has write access: https://data.neuro.polymtl.ca/datasets/ms-basel-2018

plbenveniste commented 2 months ago

I had to manually remove the files which were created eventhough dcm2niix highlighted that the input dcm files were corrupted. I didn't realize that the files were still created. They were easily identifiable thanks to the "Eq" suffix. Therefore, the following files were removed :

sub-6622_ses-20180208_acq-sagMPRAGE_T1w.json
sub-6622_ses-20180208_acq-sagMPRAGE_T1w.nii.gz
sub-6622_ses-20180208_acq-sagMPRAGE_T1w_Eq_1.nii.gz
sub-5934_ses-20171214_acq-sagMPRAGE_T1w.json
sub-5934_ses-20171214_acq-sagMPRAGE_T1w.nii.gz
sub-5934_ses-20171214_acq-sagMPRAGE_T1w_Eq_1.nii.gz

plbenveniste commented 2 months ago

PR was created and is ready for review !

plbenveniste commented 1 month ago

Suffixes were updated and modifications were pushed.

Modifications changed acq-sagMPRAGE_T1w to acq-iso_MPRAGE

plbenveniste commented 3 weeks ago

I reviewed everything based on all the conversation we have had on the file naming convention. I modified again the last modification : suffixes changed from acq-iso_MPRAGE to acq-iso_T1w.

PR is ready for review !

plbenveniste commented 1 week ago

The PR https://data.neuro.polymtl.ca/datasets/ms-basel-2018/pulls/1 was reviewed and merged ! Thanks @mguaypaq 👍

ivadomed / ms-lesion-agnostic

Create a git-annex repo for data from `basel/4031_MRI_T2_MRAGE_24patients_CGM` #19