Problem with Brain processing pipeline

[jcohenadad]

The following scripts (sent via email by @jqmcginnis to me on XXX), ran on brain scans with part of the c-spine visible (until ~C3-C4), produces an abnormally high amount of errors:

julien-macbook:~/Desktop/Brain_Cohort_Bids/derivatives/log $ ll
total 1336
drwxr-xr-x  28 julien  staff    896 24 Jun 08:27 .
drwxr-xr-x   7 julien  staff    224 24 Jun 08:29 ..
-rw-r--r--@  1 julien  staff  17182 24 Jun 08:27 err.process_brain_data_t1_sub-tum01_ses-20211217.log
-rw-r--r--   1 julien  staff  17465 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20100730.log
-rw-r--r--   1 julien  staff  19562 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20110103.log
-rw-r--r--   1 julien  staff  19824 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20120123.log
-rw-r--r--   1 julien  staff  18221 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20120614.log
-rw-r--r--   1 julien  staff  18161 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20130114.log
-rw-r--r--   1 julien  staff  17643 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20140627.log
-rw-r--r--   1 julien  staff  20179 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20150126.log
-rw-r--r--   1 julien  staff  17688 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20160125.log
-rw-r--r--   1 julien  staff  19556 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20170130.log
-rw-r--r--   1 julien  staff  19256 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20190121.log
-rw-r--r--   1 julien  staff  18924 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20200120.log
-rw-r--r--   1 julien  staff  19008 24 Jun 08:27 err.process_brain_data_t1_sub-tum02_ses-20210118.log
-rw-r--r--   1 julien  staff  17356 24 Jun 08:27 err.process_brain_data_t1_sub-tum03_ses-20180711.log
-rw-r--r--   1 julien  staff  17354 24 Jun 08:27 err.process_brain_data_t1_sub-tum04_ses-20140903.log
-rw-r--r--   1 julien  staff  16735 24 Jun 08:27 err.process_brain_data_t1_sub-tum05_ses-20090209.log
-rw-r--r--   1 julien  staff  17740 24 Jun 08:27 err.process_brain_data_t1_sub-tum07_ses-20210406.log
-rw-r--r--   1 julien  staff  17539 24 Jun 08:27 err.process_brain_data_t1_sub-tum08_ses-20120430.log
-rw-r--r--   1 julien  staff  19217 24 Jun 08:27 err.process_brain_data_t1_sub-tum08_ses-20130502.log
-rw-r--r--   1 julien  staff  21788 24 Jun 08:27 err.process_brain_data_t1_sub-tum08_ses-20140430.log
-rw-r--r--   1 julien  staff  17131 24 Jun 08:27 err.process_brain_data_t1_sub-tum08_ses-20150429.log
-rw-r--r--   1 julien  staff  45601 24 Jun 08:27 process_brain_data_t1_sub-tum02_ses-20140113.log
-rw-r--r--   1 julien  staff  52542 24 Jun 08:27 process_brain_data_t1_sub-tum02_ses-20180124.log
-rw-r--r--   1 julien  staff  55412 24 Jun 08:27 process_brain_data_t1_sub-tum04_ses-20140211.log
-rw-r--r--   1 julien  staff  78506 24 Jun 08:27 process_brain_data_t1_sub-tum06_ses-20090206.log
-rw-r--r--   1 julien  staff   6407 24 Jun 08:27 sct_run_batch_log.txt

Script `process_brain_data_t1.sh`

```console #!/bin/bash # # Process data. This script is designed to be run in the folder for a single subject, however 'sct_run_batch' can be # used to run this script multiple times in parallel across a multi-subject BIDS dataset. # # Usage: # ./process_data.sh # # Example: # ./process_data.sh sub-03 # # Author: Julien Cohen-Adad # The following global variables are retrieved from the caller sct_run_batch # but could be overwritten by uncommenting the lines below: # PATH_DATA_PROCESSED="~/data_processed" # PATH_RESULTS="~/results" # PATH_LOG="~/log" # PATH_QC="~/qc" # BASH SETTINGS # ====================================================================================================================== # Uncomment for full verbose # set -v # Immediately exit if error set -e # Exit if user presses CTRL+C (Linux) or CMD+C (OSX) trap "echo Caught Keyboard Interrupt within script. Exiting now.; exit" INT # CONVENIENCE FUNCTIONS # ====================================================================================================================== label_if_does_not_exist() { ### # This function checks if a manual label file already exists, then: # - If it does, copy it locally. # - If it doesn't, perform automatic labeling. # This allows you to add manual labels on a subject-by-subject basis without disrupting the pipeline. ### local file="$1" local file_seg="$2" # Update global variable with segmentation file name FILELABEL="${file}_seg_labeled" FILELABELMANUAL="${PATH_DATA}/derivatives/labels/${SUBJECT}/anat/${FILELABEL}-manual.nii.gz" echo "Looking for manual label: $FILELABELMANUAL" if [[ -e $FILELABELMANUAL ]]; then echo "Found! Using manual labels." rsync -avzh $FILELABELMANUAL ${FILELABEL}.nii.gz else echo "Not found. Proceeding with automatic labeling." # Generate labeled segmentation #sct_image -i ${file}.nii.gz -header #sct_image -i ${file_seg}.nii.gz -header # sct_image -i sct_image -i ${file}.nii.gz -set-sform-to-qform sct_image -i ${file_seg}.nii.gz -set-sform-to-qform sct_label_vertebrae -i ${file}.nii.gz -s ${file_seg}.nii.gz -c t1 -qc "${PATH_QC}" -qc-subject "${SUBJECT}" # Create labels in the cord at C3 and C5 mid-vertebral levels # sct_label_utils -i ${file_seg}_labeled.nii.gz -vert-body 3,5 -o ${FILELABEL}.nii.gz fi } segment_if_does_not_exist() { ### # This function checks if a manual spinal cord segmentation file already exists, then: # - If it does, copy it locally. # - If it doesn't, perform automatic spinal cord segmentation. # This allows you to add manual segmentations on a subject-by-subject basis without disrupting the pipeline. ### local file="$1" local contrast="$2" # Update global variable with segmentation file name FILESEG="${file}_seg" FILESEGMANUAL="${PATH_DATA}/derivatives/labels/${SUBJECT}/anat/${FILESEG}-manual.nii.gz" echo echo "Looking for manual segmentation: $FILESEGMANUAL" if [[ -e $FILESEGMANUAL ]]; then echo "Found! Using manual segmentation." rsync -avzh $FILESEGMANUAL ${FILESEG}.nii.gz sct_qc -i ${file}.nii.gz -s ${FILESEG}.nii.gz -p sct_deepseg_sc -qc ${PATH_QC} -qc-subject ${SUBJECT} else echo "Not found. Proceeding with automatic segmentation." # Segment spinal cord sct_deepseg_sc -i ${file}.nii.gz -c $contrast -qc ${PATH_QC} -qc-subject ${SUBJECT} fi } # SCRIPT STARTS HERE # ====================================================================================================================== # Retrieve input params IFS=/ read SUBJECT SESSION <<< $1 echo "$SUBJECT" echo "$SESSION" # get starting time: start=`date +%s` # Display useful info for the log, such as SCT version, RAM and CPU cores available sct_check_dependencies -short # Go to folder where data will be copied and processed cd $PATH_DATA_PROCESSED # Copy source images rsync -avzh $PATH_DATA/$SUBJECT . # t1ww # ====================================================================================================================== cd "${SUBJECT}/${SESSION}/anat/" file_t1w="${SUBJECT}_${SESSION}_T1w" # Segment spinal cord (only if it does not exist) segment_if_does_not_exist "${file_t1w}" "t1" file_t1w_seg="${FILESEG}" # Create labels in the cord label_if_does_not_exist "${file_t1w}" "${file_t1w_seg}" file_label="${FILELABEL}" # Register to template sct_register_to_template -i "${file_t1w}.nii.gz" -s "${file_t1w_seg}.nii.gz" -l "${file_label}.nii.gz" -c t1 \ -param step=1,type=seg,algo=centermassrot:step=2,type=im,algo=syn,iter=5,slicewise=1,metric=CC,smooth=0 \ -qc "${PATH_QC}" # Warp template # Note: we don't need the white matter atlas at this point, therefore use flag "-a 0" sct_warp_template -d "${file_t1w}.nii.gz" -w warp_template2anat.nii.gz -a 0 -ofolder label_t1w -qc "${PATH_QC}" # Compute average CSA between C1 and C2 levels (append across subjects) sct_process_segmentation -i "${file_t1w_seg}.nii.gz" -vert 1:3 -vertfile label_t1w/template/PAM50_levels.nii.gz \ -o "${PATH_RESULTS}/CSA.csv" -append 1 -qc "${PATH_QC}" # T2w # ====================================================================================================================== #cd "${SUBJECT}/anat/" #file_t2="${SUBJECT}_T2w" # Segment spinal cord (only if it does not exist) #segment_if_does_not_exist "${file_t2}" "t2" #file_t2_seg="${FILESEG}" # Create labels in the cord at C2 and C5 mid-vertebral levels (only if it does not exist) #label_if_does_not_exist "${file_t2}" "${file_t2_seg}" #file_label="${FILELABEL}" # Register to template #sct_register_to_template -i "${file_t2}.nii.gz" -s "${file_t2_seg}.nii.gz" -l "${file_label}.nii.gz" -c t2 \ # -param step=1,type=seg,algo=centermassrot:step=2,type=im,algo=syn,iter=5,slicewise=1,metric=CC,smooth=0 \ # -qc "${PATH_QC}" # Warp template # Note: we don't need the white matter atlas at this point, therefore use flag "-a 0" #sct_warp_template -d "${file_t2}.nii.gz" -w warp_template2anat.nii.gz -a 0 -ofolder label_T2w -qc "${PATH_QC}" # Compute average CSA between C2 and C3 levels (append across subjects) #sct_process_segmentation -i "${file_t2_seg}.nii.gz" -vert 2:3 -vertfile label_T2w/template/PAM50_levels.nii.gz \ # -o "${PATH_RESULTS}/CSA.csv" -append 1 -qc "${PATH_QC}" # MT # ====================================================================================================================== #file_mt1w="${SUBJECT}_acq-MTon_MTS" #file_mt0="${SUBJECT}_acq-MToff_MTS" # Segment spinal cord #segment_if_does_not_exist "${file_mt1w}" "t2s" #file_mt1w_seg=$FILESEG # Create mask #sct_create_mask -i "${file_mt1w}.nii.gz" -p centerline,"${file_mt1w_seg}.nii.gz" -size 45mm # Crop data for faster processing #sct_crop_image -i "${file_mt1w}.nii.gz" -m "mask_${file_mt1w}.nii.gz" -o "${file_mt1w}_crop.nii.gz" #sct_crop_image -i "${file_mt1w_seg}.nii.gz" -m "mask_${file_mt1w}.nii.gz" -o "${file_mt1w}_crop_seg.nii.gz" #file_mt1w="${file_mt1w}_crop" # Register mt0->mt1w # Tips: here we only use rigid transformation because both images have very # similar sequence parameters. We don't want to use SyN/BSplineSyN to avoid # introducing spurious deformations. #sct_register_multimodal -i "${file_mt0}.nii.gz" -d "${file_mt1w}.nii.gz" \ # -param step=1,type=im,algo=rigid,slicewise=1,metric=CC \ # -x spline -qc "${PATH_QC}" # Register template->mt1w # Tips: here we only use the segmentations due to poor SC/CSF contrast at the bottom slice. # Tips: First step: slicereg based on images, with large smoothing to capture # potential motion between anat and mt, then at second step: bpslinesyn in order to # adapt the shape of the cord to the mt modality (in case there are distortions between anat and mt). #sct_register_multimodal -i "${SCT_DIR}/data/PAM50/template/PAM50_t2.nii.gz" \ # -iseg "${SCT_DIR}/data/PAM50/template/PAM50_cord.nii.gz" \ # -d "${file_mt1w}.nii.gz" \ # -dseg "${file_mt1w}_seg.nii.gz" \ # -param step=1,type=seg,algo=centermass:step=2,type=seg,algo=bsplinesyn,slicewise=1,iter=3 \ # -initwarp warp_template2anat.nii.gz \ # -initwarpinv warp_anat2template.nii.gz \ # -qc "${PATH_QC}" # Rename warping fields for clarity #mv "warp_PAM50_t22${file_mt1w}.nii.gz" warp_template2mt.nii.gz #mv "warp_${file_mt1w}2PAM50_t2.nii.gz" warp_mt2template.nii.gz ## Warp template #sct_warp_template -d "${file_mt1w}.nii.gz" -w warp_template2mt.nii.gz -ofolder label_MT -qc "${PATH_QC}" # Compute mtr #sct_compute_mtr -mt0 "${file_mt0}_reg.nii.gz" -mt1w "${file_mt1w}.nii.gz" # compute MTR in dorsal columns between levels C2 and C5 (append across subjects) #sct_extract_metric -i mtr.nii.gz -f label_MT/atlas -l 53 -vert 2:5 -vertfile label_MT/template/PAM50_levels.nii.gz \ # -method map -o "${PATH_RESULTS}/MTR_in_DC.csv" -append 1 # # Verify presence of output files and write log file if error # ====================================================================================================================== FILES_TO_CHECK=( "$file_t1w_seg.nii.gz" ) for file in "${FILES_TO_CHECK[@]}"; do if [ ! -e "${file}" ]; then echo "${SUBJECT}/${file} does not exist" >> "${PATH_LOG}/error.log" fi done # Display useful info for the log end=`date +%s` runtime=$((end-start)) echo echo "~~~" echo "SCT version: `sct_version`" echo "Ran on: `uname -nsr`" echo "Duration: $(($runtime / 3600))hrs $((($runtime / 60) % 60))min $(($runtime % 60))sec" echo "~~~" ```

Example of errored log file output:

Terminal output

```console sub-tum01 ses-20211217 -- Spinal Cord Toolbox (5.6) sct_check_dependencies -short -- SCT info: - version: 5.6 - path: /home/home/johannes_julian/sct_5.6 OS: linux (Linux-5.13.0-39-generic-x86_64-with-debian-bullseye-sid) CPU cores: Available: 128, Used by ITK functions: 1 RAM: Total: 515810MB, Used: 34401MB, Available: 463628MB sending incremental file list sub-tum01/ sub-tum01/ses-20211217/ sub-tum01/ses-20211217/anat/ sub-tum01/ses-20211217/anat/sub-tum01_ses-20211217_T1w.nii.gz sent 17.05M bytes received 51 bytes 11.37M bytes/sec total size is 17.23M speedup is 1.01 Looking for manual segmentation: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/labels/sub-tum01/anat/sub-tum01_ses-20211217_T1w_seg-manual.nii.gz Not found. Proceeding with automatic segmentation. -- Spinal Cord Toolbox (5.6) sct_deepseg_sc -i sub-tum01_ses-20211217_T1w.nii.gz -c t1 -qc /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc -qc-subject sub-tum01 -- Config deepseg_sc: Centerline algorithm: svm Brain in image: True Kernel dimension: 2d Contrast: t1 Threshold: 0.15 Creating temporary folder (/tmp/sct-20220624140258.741823-t392cu5e) Reorient the image to RPI, if necessary... Finding the spinal cord centerline... Creating temporary folder (/tmp/sct-20220624140329.203542-3tcgq8s7) Remove temporary files... rm -rf /tmp/sct-20220624140329.203542-3tcgq8s7 Cropping the image around the spinal cord... Normalizing the intensity... Segmenting the spinal cord using deep learning on 2D patches... From /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. From /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.  From /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.  Reassembling the image... Resampling the segmentation to the native image resolution using linear interpolation... Binarizing the resampled segmentation... Removing small objects above slice #112 Removing small objects above slice #112 Removing small objects above slice #112 Removing small objects above slice #112 Removing small objects above slice #112 Found isolated voxels on slice 8, Removing them Remove temporary files... rm -rf /tmp/sct-20220624140258.741823-t392cu5e *** Generate Quality Control (QC) html report *** Resample images to 0.6x0.6 mm QcImage: layout with Axial slice Compute center of mass at each slice /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/sub-tum01/sub-tum01/anat/sct_deepseg_sc/2022_06_24_140419.086188/bkg_img.png Successfully generated the QC results in /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/_json/qc_2022_06_24_140419.086188.json /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3335: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /home/home/johannes_julian/sct_5.6/python/envs/venv_sct/lib/python3.7/site-packages/scipy/ndimage/measurements.py:1407: RuntimeWarning: invalid value encountered in double_scalars for dir in range(input.ndim)] To see the results in a browser, type: xdg-open /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/index.html Looking for manual label: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/labels/sub-tum01/anat/sub-tum01_ses-20211217_T1w_seg_labeled-manual.nii.gz Not found. Proceeding with automatic labeling. -- Spinal Cord Toolbox (5.6) sct_image -i sub-tum01_ses-20211217_T1w.nii.gz -set-sform-to-qform -- Generate output files... File sub-tum01_ses-20211217_T1w.nii.gz already exists. Will overwrite it. -- Spinal Cord Toolbox (5.6) sct_image -i sub-tum01_ses-20211217_T1w_seg.nii.gz -set-sform-to-qform -- Generate output files... File sub-tum01_ses-20211217_T1w_seg.nii.gz already exists. Will overwrite it. -- Spinal Cord Toolbox (5.6) sct_label_vertebrae -i sub-tum01_ses-20211217_T1w.nii.gz -s sub-tum01_ses-20211217_T1w_seg.nii.gz -c t1 -qc /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc -qc-subject sub-tum01 -- Creating temporary folder (/tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k) Copying input data to tmp folder... Straighten spinal cord... Creating temporary folder (/tmp/sct-20220624140428.949708-straighten_spinalcord-a8e8qoan) Copy files to tmp folder... Fitting centerline using B-spline approximation Error on approximation = 0.28 mm Error on approximation = 0.16 mm Error on approximation = 0.09 mm Error on approximation = 0.05 mm Error on approximation = 0.05 mm Error on approximation = 0.05 mm Create the straight space and the safe zone Length of spinal cord: 77.7393680307486 Size of spinal cord in z direction: 77.42253597479034 Ratio length/size: 1.0040922459070758 Safe zone boundaries (curved space): [-144.9935978433017, -67.57106186851136] Safe zone boundaries (straight space): [-145.15201387128081, -67.41264584053224] Pad input volume to account for spinal cord length... Time to generate centerline: 11.0 ms Warping field generated: tmp.curve2straight.nii.gz Warping field generated: tmp.straight2curve.nii.gz Apply transformation to input image... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i data.nii -o tmp.anat_rigid_warp.nii.gz -t tmp.curve2straight.nii.gz -r tmp.centerline_pad_crop.nii.gz -n 'BSpline[3]' # in /tmp/sct-20220624140428.949708-straighten_spinalcord-a8e8qoan Generate output files... File created: ./warp_curve2straight.nii.gz File created: ./warp_straight2curve.nii.gz cp /tmp/sct-20220624140428.949708-straighten_spinalcord-a8e8qoan/tmp.anat_rigid_warp.nii.gz ./straight_ref.nii.gz File created: ./data_straight.nii Remove temporary files... rm -rf /tmp/sct-20220624140428.949708-straighten_spinalcord-a8e8qoan Resample to 0.5mm isotropic... sct_resample -i data_straight.nii -mm 0.5x0.5x0.5 -x linear -o data_straightr.nii # in /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k Apply straightening to segmentation... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i segmentation.nii -o segmentation_straight.nii -t warp_curve2straight.nii.gz -r data_straightr.nii -n Linear # in /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k File /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k/segmentation_straight.nii already exists. Will overwrite it. Create label to identify disc... Creating temporary folder... Creating temporary folder (/tmp/sct-20220624140703.172690-mmr2kmio) Run C2-C3 detector... C2-C3 detected... Remove temporary files... rm -rf /tmp/sct-20220624140703.172690-mmr2kmio File /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k/labelz.nii.gz already exists. Will overwrite it. And apply straightening to label... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i labelz.nii.gz -o labelz_straight.nii.gz -t warp_curve2straight.nii.gz -r data_straightr.nii -n NearestNeighbor # in /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k Get z and disc values from straight label... .. [70, 3] Look for template... Path template: /home/home/johannes_julian/sct_5.6/data/PAM50 Open template and vertebral levels... Disc values from template: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Z-values for each disc: [963, 938, 907, 870, 833, 800, 769, 735, 692, 646, 600, 551, 500, 449, 396, 342, 289, 231, 168, 104, 79] Distances between discs (in voxel): [25.0, 31.0, 37.0, 37.0, 33.0, 31.0, 34.0, 43.0, 46.0, 46.0, 49.0, 51.0, 51.0, 53.0, 54.0, 53.0, 58.0, 63.0, 64.0, 25.0] Detect intervertebral discs... Current disc: 2 (z=70). Direction: superior .. Peak found: z=9 (correlation = 0.19984030271461697) Correlation is too low. Using adjusted template distance. Current disc: 1 (z=101). Direction: superior .. correcting factor: 1.0 .. Switching to inferior direction. Current disc: 3 (z=33). Direction: inferior Data contained zero. We probably hit the edge of the image. Correlation vector only contains zeros. Using adjusted template distance. .. correcting factor: 1.0 Adding top disc based on adjusted template distance: #0 .. approximate distance: 25 Un-straighten labeling... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i segmentation_straight_labeled.nii -o segmentation_labeled.nii -t warp_straight2curve.nii.gz -r segmentation.nii -n NearestNeighbor # in /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k Cleaning labeled segmentation: removing labeled voxels outside segmentation... Done cleaning. File /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k/segmentation_labeled.nii already exists. Will overwrite it. Label discs... Un-straighten labeled discs... /home/home/johannes_julian/sct_5.6/bin/sct_apply_transfo -i segmentation_straight_labeled_disc.nii -d segmentation.nii -w warp_straight2curve.nii.gz -o segmentation_labeled_disc.nii -x label # in /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k Generate output files... File created: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/data_processed/sub-tum01/ses-20211217/anat/sub-tum01_ses-20211217_T1w_seg_labeled.nii.gz File created: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/data_processed/sub-tum01/ses-20211217/anat/sub-tum01_ses-20211217_T1w_seg_labeled_discs.nii.gz File created: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/data_processed/sub-tum01/ses-20211217/anat/warp_curve2straight.nii.gz File created: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/data_processed/sub-tum01/ses-20211217/anat/warp_straight2curve.nii.gz File created: /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/data_processed/sub-tum01/ses-20211217/anat/straight_ref.nii.gz Remove temporary files... rm -rf /tmp/sct-20220624140427.702111-label_vertebrae-0cs7ly6k *** Generate Quality Control (QC) html report *** Resample images to NonexNone mm QcImage: layout with Sagittal slice /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/sub-tum01/sub-tum01/anat/sct_label_vertebrae/2022_06_24_140741.903467/bkg_img.png Successfully generated the QC results in /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/_json/qc_2022_06_24_140741.903467.json /home/home/johannes_julian/sct_5.6/spinalcordtoolbox/math.py:115: FutureWarning: `selem` is a deprecated argument name for `dilation`. It will be removed in version 1.0. Please use `footprint` instead. return dilation(data, selem=_get_selem(shape, size, dim), out=None) To see the results in a browser, type: xdg-open /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc/index.html -- Spinal Cord Toolbox (5.6) sct_register_to_template -i sub-tum01_ses-20211217_T1w.nii.gz -s sub-tum01_ses-20211217_T1w_seg.nii.gz -l sub-tum01_ses-20211217_T1w_seg_labeled.nii.gz -c t1 -param step=1,type=seg,algo=centermassrot:step=2,type=im,algo=syn,iter=5,slicewise=1,metric=CC,smooth=0 -qc /mnt/Drive3/julian/choroid_plexus/Brain_Cohort_Bids/derivatives/qc -- Check template files... OK: /home/home/johannes_julian/sct_5.6/data/PAM50/template/PAM50_t1.nii.gz OK: /home/home/johannes_julian/sct_5.6/data/PAM50/template/PAM50_levels.nii.gz OK: /home/home/johannes_julian/sct_5.6/data/PAM50/template/PAM50_cord.nii.gz Check parameters: Data: sub-tum01_ses-20211217_T1w.nii.gz Landmarks: sub-tum01_ses-20211217_T1w_seg_labeled.nii.gz Segmentation: sub-tum01_ses-20211217_T1w_seg.nii.gz Path template: /home/home/johannes_julian/sct_5.6/data/PAM50 Remove temp files: 1 Check input labels... ERROR: Found two labels with same value. Creating temporary folder (/tmp/sct-20220624140747.631538-register_to_template-3x1eb9do) Copying input data to tmp folder and convert to nii... Generate labels from template vertebral labeling Check if provided labels are available in the template Binarize segmentation Resample data to 1mm isotropic... load data... load data... To avoid intensity overflow due to convertion to +uint8+, intensity will be rescaled to the maximum quantization scale Change orientation of input images to RPI... Straighten the spinal cord using centerline/segmentation... Creating temporary folder (/tmp/sct-20220624140804.150012-straighten_spinalcord-mqlu3__0) Copy files to tmp folder... Create the straight space and the safe zone Length of spinal cord: 77.51278606367774 Size of spinal cord in z direction: 77.21634699164979 Ratio length/size: 1.0038390714347054 Safe zone boundaries (curved space): [-145.00261601483373, -67.78626902318395] Safe zone boundaries (straight space): [-145.15083555084772, -67.63804948716998] Pad input volume to account for spinal cord length... Time to generate centerline: 9.0 ms Warping field generated: tmp.curve2straight.nii.gz Warping field generated: tmp.straight2curve.nii.gz Apply transformation to input image... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i data.nii -o tmp.anat_rigid_warp.nii.gz -t tmp.curve2straight.nii.gz -r tmp.centerline_pad_crop.nii.gz -n 'BSpline[3]' # in /tmp/sct-20220624140804.150012-straighten_spinalcord-mqlu3__0 Generate output files... File created: ./warp_curve2straight.nii.gz File created: ./warp_straight2curve.nii.gz cp /tmp/sct-20220624140804.150012-straighten_spinalcord-mqlu3__0/tmp.anat_rigid_warp.nii.gz ./straight_ref.nii.gz File created: /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do/seg_bin_1mm_rpi_crop_straight.nii.gz Remove temporary files... rm -rf /tmp/sct-20220624140804.150012-straighten_spinalcord-mqlu3__0 /home/home/johannes_julian/sct_5.6/bin/isct_ComposeMultiTransform 3 warp_straight2curve.nii.gz -R /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do/data_1mm_rpi.nii warp_straight2curve.nii.gz # in /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do Remove unused label on template. Keep only label present in the input label image... File template_label_body.nii.gz already exists. Will overwrite it. Dilating input labels using 3vox ball radius Apply straightening to labels... Parse list of warping fields... Get dimensions of data... 200 x 252 x 252 x 1 Apply transformation... Apply transformation and resample to destination space... /home/home/johannes_julian/sct_5.6/bin/isct_antsApplyTransforms -d 3 -i /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do/label_projected_1mm_rpi_dilate.nii.gz -o /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do/label_projected_1mm_rpi_dilate_straight.nii.gz -t warp_curve2straight.nii.gz -r /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do/seg_bin_1mm_rpi_crop_straight.nii.gz -n NearestNeighbor # in /tmp/sct-20220624140747.631538-register_to_template-3x1eb9do Copy affine matrix from destination space to make sure qform/sform are the same. Estimate transformation for step #0... Labels src: [[3.983495343673397, -6.454125430616671, -75.20446811931404], [4.6454786766767455, -4.342426942110065, -92.45738015273213]] Labels dest: [[-0.0, 46.220001220703125, -86.34002685546875], [-0.0, 46.220001220703125, -100.34002685546875], [-0.0, 45.720001220703125, -117.34002685546875], [-0.0, 45.720001220703125, -135.84002685546875]] Degrees of freedom (dof): Tx_Ty_Tz_Sz /home/home/johannes_julian/sct_5.6/spinalcordtoolbox/math.py:115: FutureWarning: `selem` is a deprecated argument name for `dilation`. It will be removed in version 1.0. Please use `footprint` instead. return dilation(data, selem=_get_selem(shape, size, dim), out=None) Traceback (most recent call last): File "/home/home/johannes_julian/sct_5.6/spinalcordtoolbox/scripts/sct_register_to_template.py", line 1336, in main(sys.argv[1:]) File "/home/home/johannes_julian/sct_5.6/spinalcordtoolbox/scripts/sct_register_to_template.py", line 605, in main fname_affine='straight2templateAffine.txt', verbose=verbose) File "/home/home/johannes_julian/sct_5.6/spinalcordtoolbox/registration/landmarks.py", line 83, in register_landmarks raise Exception('Error: number of source and destination landmarks are not the same, so landmarks cannot be paired.') Exception: Error: number of source and destination landmarks are not the same, so landmarks cannot be paired. ```

[jcohenadad]

On a side note: The script is located in a folder called sct-5.5:

julien-macbook:~/Desktop/Brain_Cohort_Bids/code/sct-5.5 $ ll
total 24
drwxr-xr-x  3 julien  staff    96 24 Jun 08:29 .
drwxr-xr-x  3 julien  staff    96 24 Jun 08:29 ..
-rwxr--r--@ 1 julien  staff  9764 24 Jun 08:29 process_brain_data_t1.sh

However the log indicates that SCT v5.6 was used:

sub-tum01
ses-20211217

--
Spinal Cord Toolbox (5.6)

Uploading the script on GitHub and create releases with frozen versions of SCT (and upload log files as release assets) would minimize confusion.

EDIT 2022-07-06 23:49:47: I noticed the script was in fact uploaded to this repository https://github.com/spine-generic/spine-generic/blob/master/process_data.sh.

[jcohenadad]

Coming back to the issue, I think it is related to the fact that the registration to template done here: https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L121-L127

uses $file_label, which is the labeled segmentation instead of single-point labels at specific vertebral levels: https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L60-L64

The code that creates the single labels was commented out: https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L64

@jqmcginnis Any reason for commenting it out?

I'll uncomment it and see how the processing goes. Note that after uncommenting it, I also need to update label values form "3,5" to "2,3" because most of these scans do not include C5.

[jcohenadad]

Another issue is that the script did not seem to properly deal with sessions. The parsing was done at the beginning:

IFS=/ read SUBJECT SESSION <<< $1

But then the copying included all sessions:

rsync -avzh $PATH_DATA/$SUBJECT .

So I modified the script to:

• be agnostic to the presence of sessions or not (more generalizable)
• copy only one session per loop

Relevant chunk of the script

```console # Retrieve input params SUBJECTSESSION=$1 echo $SUBJECTSESSION # Go to folder where data will be copied and processed cd $PATH_DATA_PROCESSED # Copy source images # We use '/./' in order to include the sub-folder 'ses-0X' rsync -Ravzh $PATH_DATA/./$SUBJECTSESSION . # Go to subject folder for source images cd ${SUBJECTSESSION}/anat # Define variables # We do a substitution '/' --> '_' in case there is a subfolder 'ses-0X/' file="${SUBJECTSESSION//[\/]/_}" # T1w # ====================================================================================================================== file_t1w="${file}_T1w" ```

[jcohenadad]

another issue is that this line: https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L48

should be replaced by (see for example: https://github.com/spine-generic/spine-generic/blob/529e709501aa0c762481fc916507f74f75ca9a21/process_data.sh#L61):

  FILELABEL="${file}_labels"

Otherwise, it is referirng to the labeled segmentation, which is not the same thing as the single-point labels (needed for registration to template).

[jcohenadad]

Another issue I noticed is that, in fact, there is no need to register to the PAM50 template because the vertebral labeling can be used for computing the CSA.

[jqmcginnis]

Coming back to the issue, I think it is related to the fact that the registration to template done here:

https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L121-L127

uses $file_label, which is the labeled segmentation instead of single-point labels at specific vertebral levels:

https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L60-L64

The code that creates the single labels was commented out:

https://github.com/sct-pipeline/bavaria-quebec/blob/e648e52aad5ae2d39f70c6a15f1274011941e509/source/pipeline/brain/process_brain_data_t1.sh#L64

@jqmcginnis Any reason for commenting it out?

Thank you very much for discovering this, I confused it with the sct_label_utils --create_viewer option and thought I would be commenting out the manual labeling. However, I was commenting out the whole labeling function.

[jqmcginnis]

Thank you once again for explaining the changes and making it so clear, this really helped! Learned a lot in this issue!