search

DCAN-Labs / BIBSnet

This BIDS App provides the utility of creating a nnU-Net anatomical MRI segmentation and mask with a infant brain trained model. It can easily be included in other processing pipelines and for circumventing JLF within Nibabies.
https://bibsnet.readthedocs.io/en/latest/
Apache License 2.0
4 stars 8 forks source link

nnUNet failed to complete, exitcode -9 #137

Closed dnkennedy closed 2 months ago

dnkennedy commented 2 months ago

What happened?

Thanks for fixing my prior "Could not find a task with ID" issue. My issue has now 'migrated'.

I'm running version 3.4.2 as singularity on my University HPC: "> singularity pull bibsnet-3.4.2.sif docker://dcanumn/bibsnet:release-3.4.2"

It proceeds for awhile, and completes with an error: "ERROR 2024-09-10 23:28:28,339: nnUNet failed to complete, exitcode -9 Error: Output segmentation file not created at the path below during nnUNet_predict run. /workdir/bibsnet/sub-BB03601/ses-1/output For your input files at the path below, check their filenames and visually inspect them if needed. /workdir/bibsnet/sub-BB03601/ses-1/input"

What command did you use?

singularity run --nv --cleanenv --no-home -B /home/david.kennedy-umw/Data/Infants/BB036-BIDS:/input -B /home/david.kennedy-umw/Data/Infants/derivatives/bibsnet:/output -B /home/david.kennedy-umw/Data/Infants/workdir:/workdir /home/david.kennedy-umw/bibsnet/bibsnet-3.4.2.sif -w /workdir -sub BB03601 /input /output participant -v

What version of BIBSnet are you using?

3.4.2

Directory Structure

No response

Relevant log output

$ singularity run --nv --cleanenv --no-home -B /home/david.kennedy-umw/Data/Infants/BB036-BIDS:/input -B /home/david.kennedy-umw/Data/Infants/derivatives/bibsnet:/output -B /home/david.kennedy-umw/Data/Infants/workdir:/workdir /home/david.kennedy-umw/bibsnet/bibsnet-3.4.2.sif -w /workdir -sub BB03601 /input /output participant -v
WARNING: underlay of /etc/localtime required more than 50 (113) bind mounts
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (496) bind mounts
"uname": executable file not found in $PATH
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-f3b5vt7w because the default path (/home/david.kennedy-umw/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
VERBOSE 2024-09-10 22:12:38,088: /home/bibsnet/bibsnet -w /workdir -sub BB03601 /input /output participant -v
VERBOSE 2024-09-10 22:12:38,090: All parameters from input args:
{'common': {'fsl_bin_path': '/opt/fsl-6.0.5.1/bin', 'bids_dir': '/input', 'overwrite': False, 'work_dir': '/workdir'}, 'bibsnet': {'model': '3d_fullres', 'nnUNet_predict_path': '/opt/conda/bin/nnUNet_predict'}, 'stage_names': {'start': 'prebibsnet', 'end': 'postbibsnet'}, 'optional_out_dirs': {'prebibsnet': '/workdir/prebibsnet', 'bibsnet': '/workdir/bibsnet', 'postbibsnet': '/workdir/postbibsnet', 'derivatives': '/output'}}
VERBOSE 2024-09-10 22:12:38,090: All required input files exist.
INFO 2024-09-10 22:12:38,090: Now running prebibsnet stage on:
{'subject': 'sub-BB03601', 'session': 'ses-1', 'has_T1w': True, 'has_T2w': True, 'model': 540}
INFO 2024-09-10 22:12:38,363: Now running FSL FLIRT:
flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-21_T1w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-10_T1w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_acq-MRPAGE_run-10_desc-avg_T1w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1_avg.mat -bins 640 -searchcost mutualinfo
INFO 2024-09-10 22:15:28,521: Now running FSL FLIRT:
flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-14_T1w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-10_T1w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_acq-MRPAGE_run-10_desc-avg_T1w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1_avg.mat -bins 640 -searchcost mutualinfo
INFO 2024-09-10 22:19:28,824: Now running FSL FLIRT:
flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-20_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo
INFO 2024-09-10 22:36:38,997: Now running FSL FLIRT:
flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-11_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo
INFO 2024-09-10 22:53:39,981: Now running FSL FLIRT:
flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-13_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo
INFO 2024-09-10 23:11:02,071: Denoising input avg image
240910-23:11:04,716 nipype.workflow INFO:
     Workflow T1w_denoise_and_bfcorrect settings: ['check', 'execution', 'logging', 'monitoring']
240910-23:11:04,771 nipype.workflow INFO:
     Running serially.
240910-23:11:04,771 nipype.workflow INFO:
     [Node] Setting-up "T1w_denoise_and_bfcorrect.clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/clip".
240910-23:11:04,772 nipype.workflow INFO:
     [Node] Outdated cache found for "T1w_denoise_and_bfcorrect.clip".
240910-23:11:04,837 nipype.workflow INFO:
     [Node] Executing "clip" <niworkflows.interfaces.nibabel.IntensityClip>
240910-23:11:12,172 nipype.workflow INFO:
     [Node] Finished "clip", elapsed time 7.332913s.
240910-23:11:12,194 nipype.workflow INFO:
     [Node] Setting-up "T1w_denoise_and_bfcorrect.denoise" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/denoise".
240910-23:11:12,196 nipype.workflow INFO:
     [Node] Outdated cache found for "T1w_denoise_and_bfcorrect.denoise".
240910-23:11:12,259 nipype.workflow INFO:
     [Node] Executing "denoise" <nipype.interfaces.ants.segmentation.DenoiseImage>
240910-23:12:05,165 nipype.workflow INFO:
     [Node] Finished "denoise", elapsed time 52.904243s.
240910-23:12:05,187 nipype.workflow INFO:
     [Node] Setting-up "T1w_denoise_and_bfcorrect.n4_correct" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/n4_correct".
240910-23:12:05,189 nipype.workflow INFO:
     [Node] Outdated cache found for "T1w_denoise_and_bfcorrect.n4_correct".
240910-23:12:05,261 nipype.workflow INFO:
     [Node] Executing "n4_correct" <nipype.interfaces.ants.segmentation.N4BiasFieldCorrection>
240910-23:13:02,233 nipype.workflow INFO:
     [Node] Finished "n4_correct", elapsed time 56.970625s.
240910-23:13:02,255 nipype.workflow INFO:
     [Node] Setting-up "T1w_denoise_and_bfcorrect.final_clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/final_clip".
240910-23:13:02,256 nipype.workflow INFO:
     [Node] Outdated cache found for "T1w_denoise_and_bfcorrect.final_clip".
240910-23:13:02,326 nipype.workflow INFO:
     [Node] Executing "final_clip" <niworkflows.interfaces.nibabel.IntensityClip>
240910-23:13:06,424 nipype.workflow INFO:
     [Node] Finished "final_clip", elapsed time 4.096543s.
INFO 2024-09-10 23:13:06,474: Denoising input avg image
240910-23:13:06,476 nipype.workflow INFO:
     Workflow T2w_denoise_and_bfcorrect settings: ['check', 'execution', 'logging', 'monitoring']
240910-23:13:06,521 nipype.workflow INFO:
     Running serially.
240910-23:13:06,521 nipype.workflow INFO:
     [Node] Setting-up "T2w_denoise_and_bfcorrect.clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/clip".
240910-23:13:06,522 nipype.workflow INFO:
     [Node] Outdated cache found for "T2w_denoise_and_bfcorrect.clip".
240910-23:13:06,581 nipype.workflow INFO:
     [Node] Executing "clip" <niworkflows.interfaces.nibabel.IntensityClip>
240910-23:13:45,542 nipype.workflow INFO:
     [Node] Finished "clip", elapsed time 38.958652s.
240910-23:13:45,563 nipype.workflow INFO:
     [Node] Setting-up "T2w_denoise_and_bfcorrect.denoise" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/denoise".
240910-23:13:45,565 nipype.workflow INFO:
     [Node] Outdated cache found for "T2w_denoise_and_bfcorrect.denoise".
240910-23:13:45,664 nipype.workflow INFO:
     [Node] Executing "denoise" <nipype.interfaces.ants.segmentation.DenoiseImage>
240910-23:20:57,53 nipype.workflow INFO:
     [Node] Finished "denoise", elapsed time 431.386766s.
240910-23:20:57,90 nipype.workflow INFO:
     [Node] Setting-up "T2w_denoise_and_bfcorrect.n4_correct" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/n4_correct".
240910-23:20:57,92 nipype.workflow INFO:
     [Node] Outdated cache found for "T2w_denoise_and_bfcorrect.n4_correct".
240910-23:20:57,143 nipype.workflow INFO:
     [Node] Executing "n4_correct" <nipype.interfaces.ants.segmentation.N4BiasFieldCorrection>
240910-23:26:37,750 nipype.workflow INFO:
     [Node] Finished "n4_correct", elapsed time 340.604846s.
240910-23:26:37,792 nipype.workflow INFO:
     [Node] Setting-up "T2w_denoise_and_bfcorrect.final_clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/final_clip".
240910-23:26:37,794 nipype.workflow INFO:
     [Node] Outdated cache found for "T2w_denoise_and_bfcorrect.final_clip".
240910-23:26:37,845 nipype.workflow INFO:
     [Node] Executing "final_clip" <niworkflows.interfaces.nibabel.IntensityClip>
240910-23:27:05,35 nipype.workflow INFO:
     [Node] Finished "final_clip", elapsed time 27.187524s.
Configuring model on the CPU
Running SynthStrip model version 1
Input image read from: /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0000.nii.gz
Masked image saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/skullstripped.nii.gz
Binary brain mask saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/brainmask.nii.gz
If you use SynthStrip in your analysis, please cite:
----------------------------------------------------
SynthStrip: Skull-Stripping for Any Brain Image.
A Hoopes, JS Mora, AV Dalca, B Fischl, M Hoffmann.
Now cropping average image at z-coordinate plane 30
Configuring model on the CPU
Running SynthStrip model version 1
Input image read from: /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0001.nii.gz
Masked image saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/skullstripped.nii.gz
Binary brain mask saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/brainmask.nii.gz
If you use SynthStrip in your analysis, please cite:
----------------------------------------------------
SynthStrip: Skull-Stripping for Any Brain Image.
A Hoopes, JS Mora, AV Dalca, B Fischl, M Hoffmann.
Now cropping average image at z-coordinate plane 62
INFO 2024-09-10 23:27:30,399: The anatomical images have been cropped for use in BIBSnet
VERBOSE 2024-09-10 23:27:30,399: Input images for T1w registration:
T1w: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz
T2w: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz
VERBOSE 2024-09-10 23:27:30,401: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_to_BIBS.nii.gz
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T1w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,401: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_to_BIBS.nii.gz
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T1w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,401: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/cropT2tocropT1.mat
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_registered_to_T1w.nii.gz
VERBOSE 2024-09-10 23:27:30,401: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_to_BIBS.nii.gz
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T2w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,401: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_to_BIBS.nii.gz
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T2w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,401: Arguments for non-ACPC image transformation:
{'out_dir': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms', 'resolution': '1', 'ident_mx': '/home/bibsnet/data/identity_matrix.mat', 'ref_img': '/home/bibsnet/data/MNI_templates/INFANT_MNI_T{}_1mm.nii.gz', 'reg_input_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz', 'output_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_registered_to_T1w.nii.gz', 'reg_input_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz', 'output_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_registered_to_T1w.nii.gz'}
VERBOSE 2024-09-10 23:27:30,403: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_crop2acpc.mat
VERBOSE 2024-09-10 23:27:30,403: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat
VERBOSE 2024-09-10 23:27:30,404: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat
VERBOSE 2024-09-10 23:27:30,404: Now running FSL command:
/opt/fsl-6.0.5.1/bin/aff2rigid /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat -v
VERBOSE 2024-09-10 23:27:30,591: FSL command completed
VERBOSE 2024-09-10 23:27:30,592: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz
VERBOSE 2024-09-10 23:27:30,592: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_crop2acpc.mat
VERBOSE 2024-09-10 23:27:30,592: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat
VERBOSE 2024-09-10 23:27:30,592: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat
VERBOSE 2024-09-10 23:27:30,592: Now running FSL command:
/opt/fsl-6.0.5.1/bin/aff2rigid /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat -v
VERBOSE 2024-09-10 23:27:30,769: FSL command completed
VERBOSE 2024-09-10 23:27:30,769: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz
VERBOSE 2024-09-10 23:27:30,769: Input images for T1w registration:
T1w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz
T2w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/cropT2tocropT1.mat
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T2w_registered_to_T1w.nii.gz
VERBOSE 2024-09-10 23:27:30,823: Arguments for ACPC image transformation:
{'out_dir': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align', 'resolution': '1', 'ident_mx': '/home/bibsnet/data/identity_matrix.mat', 'ref_img': '/home/bibsnet/data/MNI_templates/INFANT_MNI_T{}_1mm.nii.gz', 'reg_input_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz', 'output_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T1w_registered_to_T1w.nii.gz', 'reg_input_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz', 'output_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T2w_registered_to_T1w.nii.gz', 'crop_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz', 'crop_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz', 'mats_T1w': {'crop2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_crop2acpc.mat', 'full2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat', 'full2crop': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat', 'acpc2rigidbody': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat'}, 'mats_T2w': {'crop2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_crop2acpc.mat', 'full2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat', 'full2crop': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat', 'acpc2rigidbody': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat'}}
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/preBIBSnet_final_0000.nii.gz
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT1w.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T1w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0000.nii.gz
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_to_rigidbody.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/preBIBSnet_final_0001.nii.gz
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2w.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2toT1.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL convert_xfm command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T2w_to_BIBS_template.mat
VERBOSE 2024-09-10 23:27:30,823: Skipping FSL applywarp command because its output image(s) listed below exist(s) and overwrite=False.
/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0001.nii.gz
VERBOSE 2024-09-10 23:27:30,823: ACPC:
VERBOSE 2024-09-10 23:27:31,103: T1w has 18145 negatives.
VERBOSE 2024-09-10 23:27:31,320: T2w has 1456 negatives.
VERBOSE 2024-09-10 23:27:33,294: 
Vectors: {'T1w': array([0., 0., 0., ..., 0., 0., 0.]), 'T2w': array([0., 0., 0., ..., 0., 0., 0.])}
Mean Within: [0. 0. 0. ... 0. 0. 0.]
Mean Total: 21.908873967045153
SumSq Within: 9666687902.0
SumSq Total: 40464187639.63406
VERBOSE 2024-09-10 23:27:33,295: Non-ACPC:
VERBOSE 2024-09-10 23:27:33,445: T1w has 370 negatives.
VERBOSE 2024-09-10 23:27:33,593: T2w has 1332 negatives.
VERBOSE 2024-09-10 23:27:35,559: 
Vectors: {'T1w': array([0., 0., 0., ..., 0., 0., 0.]), 'T2w': array([0., 0., 0., ..., 0., 0., 0.])}
Mean Within: [0. 0. 0. ... 0. 0. 0.]
Mean Total: 14.696409460586796
SumSq Within: 4715746430.0
SumSq Total: 28676188898.99348
VERBOSE 2024-09-10 23:27:35,560: Eta-Squared Values: {'ACPC': 0.7611051038985489, 'non-ACPC': 0.835551842449834}
INFO 2024-09-10 23:27:35,560: Using only T2w-to-T1w registration for resizing.
VERBOSE 2024-09-10 23:27:35,560: 
T1w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0000.nii.gz
T2w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0001.nii.gz
INFO 2024-09-10 23:27:35,560: The anatomical images have been resized for use in BIBSnet
INFO 2024-09-10 23:27:35,609: prebibsnet finished on subject sub-BB03601 session ses-1. Time elapsed since prebibsnet started: 1:14:57.518610
INFO 2024-09-10 23:27:35,609: Now running bibsnet stage on:
{'subject': 'sub-BB03601', 'session': 'ses-1', 'has_T1w': True, 'has_T2w': True, 'model': 540}
VERBOSE 2024-09-10 23:27:35,610: Now running BIBSnet with these parameters:
{'model': '3d_fullres', 'nnUNet': '/opt/conda/bin/nnUNet_predict', 'input': '/workdir/bibsnet/sub-BB03601/ses-1/input', 'output': '/workdir/bibsnet/sub-BB03601/ses-1/output', 'task': '540'}
VERBOSE 2024-09-10 23:27:35,611: Now running nnUNet with these parameters: ['/opt/conda/bin/nnUNet_predict', '-i', '/workdir/bibsnet/sub-BB03601/ses-1/input', '-o', '/workdir/bibsnet/sub-BB03601/ses-1/output', '-t', '540', '-m', '3d_fullres', '--disable_tta']
ERROR 2024-09-10 23:28:28,339: nnUNet failed to complete, exitcode -9
Error: Output segmentation file not created at the path below during nnUNet_predict run.
/workdir/bibsnet/sub-BB03601/ses-1/output
For your input files at the path below, check their filenames and visually inspect them if needed.
/workdir/bibsnet/sub-BB03601/ses-1/input

Add any additional information or context about the problem here.

It seems to have finished the prebibsnet stage. In the bibsnet part of the workingdir, input, I have two images, sub-BB03601_ses-1_optimal_resized_0000.nii.gz sub-BB03601_ses-1_optimal_resized_0001.nii.gz

I've attempted to add a snapshot of each. screenshot_T1 screenshot_T2

The centering of these seem, let's say, unusual...

LuciMoore commented 2 months ago

@dnkennedy the images look over-cropped. This should have been fixed with the latest updates, but clearly doesn't seem to be working for this data. It's not immediately apparent to me why that is, but I have a couple of ideas. It would be awesome if you could share the input and/or output data with me so I can take a closer look and do some troubleshooting on our end, but I understand if that's not be permissible without a DUA

From the logs, it looks like the work dir contains old outputs (eg Skipping FSL flirt command because its output image(s) listed below exist(s)), so you could try wiping the outputs and re-running as a sanity check in case these intermediate outputs were generated by a prior version (you could also try the --overwrite flag, but I think it's buggy (see #44 ), so probably safer to just start fresh). We could also work on providing test data soon for an additional sanity check (issue #135)

I'll look at the info you've provided more closely in the next few days, but let me know if you're able to share the data for this subject!

dnkennedy commented 2 months ago

@LuciMoore thanks for looking at this. I cleared the workdir, and reran. Same error (I think) but without the "skipping FLIRT" notifications. The centering of the prebidsnet looks better. (snapshots included below). I'm looking into the sharing status for this data...

BIBSNet call and output ``` $ singularity run --nv --cleanenv --no-home -B /home/david.kennedy-umw/Data/Infants/BB036-BIDS:/input -B /home/david.kennedy-umw/Data/Infants/derivatives/bibsnet:/output -B /home/david.kennedy-umw/Data/Infants/workdir:/workdir /home/david.kennedy-umw/bibsnet/bibsnet-3.4.2.sif -w /workdir -sub BB03601 /input /output participant -v WARNING: underlay of /etc/localtime required more than 50 (113) bind mounts WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (496) bind mounts "uname": executable file not found in $PATH Matplotlib created a temporary config/cache directory at /tmp/matplotlib-fmr343dy because the default path (/home/david.kennedy-umw/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. VERBOSE 2024-09-13 11:24:34,196: /home/bibsnet/bibsnet -w /workdir -sub BB03601 /input /output participant -v VERBOSE 2024-09-13 11:24:34,198: All parameters from input args: {'common': {'fsl_bin_path': '/opt/fsl-6.0.5.1/bin', 'bids_dir': '/input', 'overwrite': False, 'work_dir': '/workdir'}, 'bibsnet': {'model': '3d_fullres', 'nnUNet_predict_path': '/opt/conda/bin/nnUNet_predict'}, 'stage_names': {'start': 'prebibsnet', 'end': 'postbibsnet'}, 'optional_out_dirs': {'prebibsnet': '/workdir/prebibsnet', 'bibsnet': '/workdir/bibsnet', 'postbibsnet': '/workdir/postbibsnet', 'derivatives': '/output'}} VERBOSE 2024-09-13 11:24:34,198: All required input files exist. INFO 2024-09-13 11:24:34,198: Now running prebibsnet stage on: {'subject': 'sub-BB03601', 'session': 'ses-1', 'has_T1w': True, 'has_T2w': True, 'model': 540} INFO 2024-09-13 11:24:34,636: Now running FSL FLIRT: flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-21_T1w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-10_T1w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_acq-MRPAGE_run-10_desc-avg_T1w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1_avg.mat -bins 640 -searchcost mutualinfo INFO 2024-09-13 11:27:24,776: Now running FSL FLIRT: flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-14_T1w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_acq-MRPAGE_run-10_T1w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_acq-MRPAGE_run-10_desc-avg_T1w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1_avg.mat -bins 640 -searchcost mutualinfo INFO 2024-09-13 11:31:36,140: Now running FSL FLIRT: flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-20_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo INFO 2024-09-13 11:50:09,923: Now running FSL FLIRT: flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-11_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo INFO 2024-09-13 12:08:10,578: Now running FSL FLIRT: flirt -in /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-13_T2w.nii.gz -ref /input/sub-BB03601/ses-1/anat/sub-BB03601_ses-1_run-12_T2w.nii.gz -out /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_run-12_desc-avg_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2_avg.mat -bins 640 -searchcost mutualinfo INFO 2024-09-13 12:26:38,104: Denoising input avg image 240913-12:26:41,970 nipype.workflow INFO: Workflow T1w_denoise_and_bfcorrect settings: ['check', 'execution', 'logging', 'monitoring'] 240913-12:26:42,192 nipype.workflow INFO: Running serially. 240913-12:26:42,192 nipype.workflow INFO: [Node] Setting-up "T1w_denoise_and_bfcorrect.clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/clip". 240913-12:26:42,364 nipype.workflow INFO: [Node] Executing "clip" 240913-12:26:51,95 nipype.workflow INFO: [Node] Finished "clip", elapsed time 8.729331s. 240913-12:26:51,175 nipype.workflow INFO: [Node] Setting-up "T1w_denoise_and_bfcorrect.denoise" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/denoise". 240913-12:26:51,268 nipype.workflow INFO: [Node] Executing "denoise" 240913-12:27:47,352 nipype.workflow INFO: [Node] Finished "denoise", elapsed time 56.081771s. 240913-12:27:47,415 nipype.workflow INFO: [Node] Setting-up "T1w_denoise_and_bfcorrect.n4_correct" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/n4_correct". 240913-12:27:47,643 nipype.workflow INFO: [Node] Executing "n4_correct" 240913-12:28:47,844 nipype.workflow INFO: [Node] Finished "n4_correct", elapsed time 60.199164s. 240913-12:28:47,959 nipype.workflow INFO: [Node] Setting-up "T1w_denoise_and_bfcorrect.final_clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T1w_denoise_and_bfcorrect/final_clip". 240913-12:28:48,101 nipype.workflow INFO: [Node] Executing "final_clip" 240913-12:28:52,527 nipype.workflow INFO: [Node] Finished "final_clip", elapsed time 4.424234s. INFO 2024-09-13 12:28:52,668: Denoising input avg image 240913-12:28:52,670 nipype.workflow INFO: Workflow T2w_denoise_and_bfcorrect settings: ['check', 'execution', 'logging', 'monitoring'] 240913-12:28:52,888 nipype.workflow INFO: Running serially. 240913-12:28:52,888 nipype.workflow INFO: [Node] Setting-up "T2w_denoise_and_bfcorrect.clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/clip". 240913-12:28:53,124 nipype.workflow INFO: [Node] Executing "clip" 240913-12:29:35,82 nipype.workflow INFO: [Node] Finished "clip", elapsed time 41.956744s. 240913-12:29:35,137 nipype.workflow INFO: [Node] Setting-up "T2w_denoise_and_bfcorrect.denoise" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/denoise". 240913-12:29:35,250 nipype.workflow INFO: [Node] Executing "denoise" 240913-12:37:04,397 nipype.workflow INFO: [Node] Finished "denoise", elapsed time 449.1446s. 240913-12:37:04,435 nipype.workflow INFO: [Node] Setting-up "T2w_denoise_and_bfcorrect.n4_correct" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/n4_correct". 240913-12:37:04,769 nipype.workflow INFO: [Node] Executing "n4_correct" 240913-12:43:02,103 nipype.workflow INFO: [Node] Finished "n4_correct", elapsed time 357.331609s. 240913-12:43:02,143 nipype.workflow INFO: [Node] Setting-up "T2w_denoise_and_bfcorrect.final_clip" in "/workdir/prebibsnet/sub-BB03601/ses-1/averaged/T2w_denoise_and_bfcorrect/final_clip". 240913-12:43:02,505 nipype.workflow INFO: [Node] Executing "final_clip" 240913-12:43:31,589 nipype.workflow INFO: [Node] Finished "final_clip", elapsed time 29.081853s. Configuring model on the CPU Running SynthStrip model version 1 Input image read from: /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0000.nii.gz Masked image saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/skullstripped.nii.gz Binary brain mask saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/brainmask.nii.gz If you use SynthStrip in your analysis, please cite: ---------------------------------------------------- SynthStrip: Skull-Stripping for Any Brain Image. A Hoopes, JS Mora, AV Dalca, B Fischl, M Hoffmann. Now cropping average image at z-coordinate plane 30 Configuring model on the CPU Running SynthStrip model version 1 Input image read from: /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0001.nii.gz Masked image saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/skullstripped.nii.gz Binary brain mask saved to: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/brainmask.nii.gz If you use SynthStrip in your analysis, please cite: ---------------------------------------------------- SynthStrip: Skull-Stripping for Any Brain Image. A Hoopes, JS Mora, AV Dalca, B Fischl, M Hoffmann. Now cropping average image at z-coordinate plane 62 INFO 2024-09-13 12:44:07,078: The anatomical images have been cropped for use in BIBSnet VERBOSE 2024-09-13 12:44:07,165: Input images for T1w registration: T1w: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz T2w: /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz VERBOSE 2024-09-13 12:44:07,166: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -in /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz -ref /home/bibsnet/data/MNI_templates/INFANT_MNI_T1_1mm.nii.gz -applyisoxfm 1 -init /home/bibsnet/data/identity_matrix.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_to_BIBS.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T1w_to_BIBS_template.mat -v VERBOSE 2024-09-13 12:44:09,594: FSL command completed VERBOSE 2024-09-13 12:44:09,595: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False. /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_to_BIBS.nii.gz /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T1w_to_BIBS_template.mat VERBOSE 2024-09-13 12:44:09,595: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -ref /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz -in /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/cropT2tocropT1.mat -out /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_registered_to_T1w.nii.gz -cost mutualinfo -searchrx -15 15 -searchry -15 15 -searchrz -15 15 -dof 6 -v VERBOSE 2024-09-13 12:48:58,833: FSL command completed VERBOSE 2024-09-13 12:48:58,834: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -in /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_registered_to_T1w.nii.gz -ref /home/bibsnet/data/MNI_templates/INFANT_MNI_T2_1mm.nii.gz -applyisoxfm 1 -init /home/bibsnet/data/identity_matrix.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_to_BIBS.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T2w_to_BIBS_template.mat -v WARNING: Both reference and input images have an sform matrix set VERBOSE 2024-09-13 12:49:00,789: FSL command completed VERBOSE 2024-09-13 12:49:00,790: Skipping FSL flirt command because its output image(s) listed below exist(s) and overwrite=False. /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_to_BIBS.nii.gz /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T2w_to_BIBS_template.mat VERBOSE 2024-09-13 12:49:00,790: Arguments for non-ACPC image transformation: {'out_dir': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms', 'resolution': '1', 'ident_mx': '/home/bibsnet/data/identity_matrix.mat', 'ref_img': '/home/bibsnet/data/MNI_templates/INFANT_MNI_T{}_1mm.nii.gz', 'reg_input_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz', 'output_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T1w_registered_to_T1w.nii.gz', 'reg_input_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz', 'output_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/T2w_registered_to_T1w.nii.gz'} VERBOSE 2024-09-13 12:49:00,793: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -interp spline -ref /home/bibsnet/data/MNI_templates/INFANT_MNI_T1_1mm.nii.gz -in /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_crop2acpc.mat -searchrx -45 45 -searchry -30 30 -searchrz -30 30 -v VERBOSE 2024-09-13 12:49:28,246: FSL command completed VERBOSE 2024-09-13 12:49:28,246: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -inverse /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/crop2full.mat -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat -v VERBOSE 2024-09-13 12:49:28,736: FSL command completed VERBOSE 2024-09-13 12:49:28,736: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_crop2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat -v VERBOSE 2024-09-13 12:49:28,752: FSL command completed VERBOSE 2024-09-13 12:49:28,752: Now running FSL command: /opt/fsl-6.0.5.1/bin/aff2rigid /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat -v VERBOSE 2024-09-13 12:49:29,032: FSL command completed VERBOSE 2024-09-13 12:49:29,032: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0000.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T1_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz -v VERBOSE 2024-09-13 12:49:34,131: FSL command completed VERBOSE 2024-09-13 12:49:34,131: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -interp spline -ref /home/bibsnet/data/MNI_templates/INFANT_MNI_T2_1mm.nii.gz -in /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_crop2acpc.mat -searchrx -45 45 -searchry -30 30 -searchrz -30 30 -v VERBOSE 2024-09-13 12:53:10,038: FSL command completed VERBOSE 2024-09-13 12:53:10,038: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -inverse /workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/crop2full.mat -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat -v VERBOSE 2024-09-13 12:53:10,053: FSL command completed VERBOSE 2024-09-13 12:53:10,053: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_crop2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat -v VERBOSE 2024-09-13 12:53:10,066: FSL command completed VERBOSE 2024-09-13 12:53:10,066: Now running FSL command: /opt/fsl-6.0.5.1/bin/aff2rigid /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat -v VERBOSE 2024-09-13 12:53:10,258: FSL command completed VERBOSE 2024-09-13 12:53:10,259: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0001.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T2_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz -v VERBOSE 2024-09-13 12:53:17,423: FSL command completed VERBOSE 2024-09-13 12:53:17,423: Input images for T1w registration: T1w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz T2w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz VERBOSE 2024-09-13 12:53:17,800: Now running FSL command: /opt/fsl-6.0.5.1/bin/flirt -ref /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz -in /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/cropT2tocropT1.mat -out /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T2w_registered_to_T1w.nii.gz -cost mutualinfo -searchrx -15 15 -searchry -15 15 -searchrz -15 15 -dof 6 -v WARNING: Both reference and input images have an sform matrix set VERBOSE 2024-09-13 12:54:35,148: FSL command completed VERBOSE 2024-09-13 12:54:35,148: Arguments for ACPC image transformation: {'out_dir': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align', 'resolution': '1', 'ident_mx': '/home/bibsnet/data/identity_matrix.mat', 'ref_img': '/home/bibsnet/data/MNI_templates/INFANT_MNI_T{}_1mm.nii.gz', 'reg_input_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T1w.nii.gz', 'output_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T1w_registered_to_T1w.nii.gz', 'reg_input_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_aligned_T2w.nii.gz', 'output_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/ACPC_T2w_registered_to_T1w.nii.gz', 'crop_T1w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T1w/sub-BB03601_ses-1_0000.nii.gz', 'crop_T2w_img': '/workdir/prebibsnet/sub-BB03601/ses-1/cropped/T2w/sub-BB03601_ses-1_0001.nii.gz', 'mats_T1w': {'crop2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_crop2acpc.mat', 'full2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2acpc.mat', 'full2crop': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat', 'acpc2rigidbody': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat'}, 'mats_T2w': {'crop2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_crop2acpc.mat', 'full2acpc': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2acpc.mat', 'full2crop': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat', 'acpc2rigidbody': '/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat'}} VERBOSE 2024-09-13 12:54:35,157: Copying /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_acpc2rigidbody.mat to /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/crop_T1w_to_BIBS_template.mat VERBOSE 2024-09-13 12:54:35,157: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0000.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T1_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/crop_T1w_to_BIBS_template.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/preBIBSnet_final_0000.nii.gz -v VERBOSE 2024-09-13 12:54:39,540: FSL command completed VERBOSE 2024-09-13 12:54:39,540: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT1w.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T1w_full2crop.mat /home/bibsnet/data/identity_matrix.mat -v VERBOSE 2024-09-13 12:54:39,556: FSL command completed VERBOSE 2024-09-13 12:54:39,557: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T1w_to_BIBS_template.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT1w.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T1w_to_BIBS_template.mat -v VERBOSE 2024-09-13 12:54:39,569: FSL command completed VERBOSE 2024-09-13 12:54:39,570: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0000.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T1_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T1w_to_BIBS_template.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0000.nii.gz -v VERBOSE 2024-09-13 12:54:43,824: FSL command completed VERBOSE 2024-09-13 12:54:43,824: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_to_rigidbody.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/cropT2tocropT1.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_acpc2rigidbody.mat -v VERBOSE 2024-09-13 12:54:43,839: FSL command completed VERBOSE 2024-09-13 12:54:43,847: Copying /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_to_rigidbody.mat to /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/crop_T2w_to_BIBS_template.mat VERBOSE 2024-09-13 12:54:43,847: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0001.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T2_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/crop_T2w_to_BIBS_template.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/preBIBSnet_final_0001.nii.gz -v VERBOSE 2024-09-13 12:54:51,395: FSL command completed VERBOSE 2024-09-13 12:54:51,396: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2w.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/ACPC_align/T2w_full2crop.mat /home/bibsnet/data/identity_matrix.mat -v VERBOSE 2024-09-13 12:54:51,412: FSL command completed VERBOSE 2024-09-13 12:54:51,412: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2toT1.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/cropT2tocropT1.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2w.mat -v VERBOSE 2024-09-13 12:54:51,425: FSL command completed VERBOSE 2024-09-13 12:54:51,425: Now running FSL command: /opt/fsl-6.0.5.1/bin/convert_xfm -omat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T2w_to_BIBS_template.mat -concat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/crop_T2w_to_BIBS_template.mat /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full2cropT2toT1.mat -v VERBOSE 2024-09-13 12:54:51,441: FSL command completed VERBOSE 2024-09-13 12:54:51,441: Now running FSL command: /opt/fsl-6.0.5.1/bin/applywarp --rel --interp=spline -i /workdir/prebibsnet/sub-BB03601/ses-1/averaged/sub-BB03601_ses-1_0001.nii.gz -r /home/bibsnet/data/MNI_templates/INFANT_MNI_T2_1mm.nii.gz --premat=/workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T2w_to_BIBS_template.mat -o /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0001.nii.gz -v VERBOSE 2024-09-13 12:54:57,793: FSL command completed VERBOSE 2024-09-13 12:54:57,794: ACPC: VERBOSE 2024-09-13 12:54:57,921: T1w has 18304 negatives. VERBOSE 2024-09-13 12:54:58,042: T2w has 1509 negatives. VERBOSE 2024-09-13 12:55:00,244: Vectors: {'T1w': array([0., 0., 0., ..., 0., 0., 0.]), 'T2w': array([0., 0., 0., ..., 0., 0., 0.])} Mean Within: [0. 0. 0. ... 0. 0. 0.] Mean Total: 22.147409400761553 SumSq Within: 9848948792.0 SumSq Total: 40781774075.971176 VERBOSE 2024-09-13 12:55:00,245: Non-ACPC: VERBOSE 2024-09-13 12:55:00,367: T1w has 373 negatives. VERBOSE 2024-09-13 12:55:00,488: T2w has 1468 negatives. VERBOSE 2024-09-13 12:55:02,649: Vectors: {'T1w': array([0., 0., 0., ..., 0., 0., 0.]), 'T2w': array([0., 0., 0., ..., 0., 0., 0.])} Mean Within: [0. 0. 0. ... 0. 0. 0.] Mean Total: 20.72497206770445 SumSq Within: 8657205400.5 SumSq Total: 38723365127.92024 VERBOSE 2024-09-13 12:55:02,650: Eta-Squared Values: {'ACPC': 0.7584963132390298, 'non-ACPC': 0.7764345796936434} INFO 2024-09-13 12:55:02,650: Using only T2w-to-T1w registration for resizing. VERBOSE 2024-09-13 12:55:02,650: T1w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0000.nii.gz T2w: /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/preBIBSnet_final_0001.nii.gz INFO 2024-09-13 12:55:02,650: The anatomical images have been resized for use in BIBSnet VERBOSE 2024-09-13 12:55:02,860: Copying /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T1w_to_BIBS_template.mat to /workdir/postbibsnet/sub-BB03601/ses-1/preBIBSnet_full_crop_T1w_to_BIBS_template.mat VERBOSE 2024-09-13 12:55:02,909: Copying /workdir/prebibsnet/sub-BB03601/ses-1/resized/xfms/full_crop_T2w_to_BIBS_template.mat to /workdir/postbibsnet/sub-BB03601/ses-1/preBIBSnet_full_crop_T2w_to_BIBS_template.mat INFO 2024-09-13 12:55:02,910: prebibsnet finished on subject sub-BB03601 session ses-1. Time elapsed since prebibsnet started: 1:30:28.711913 INFO 2024-09-13 12:55:02,910: Now running bibsnet stage on: {'subject': 'sub-BB03601', 'session': 'ses-1', 'has_T1w': True, 'has_T2w': True, 'model': 540} VERBOSE 2024-09-13 12:55:02,913: Now running BIBSnet with these parameters: {'model': '3d_fullres', 'nnUNet': '/opt/conda/bin/nnUNet_predict', 'input': '/workdir/bibsnet/sub-BB03601/ses-1/input', 'output': '/workdir/bibsnet/sub-BB03601/ses-1/output', 'task': '540'} VERBOSE 2024-09-13 12:55:02,913: Now running nnUNet with these parameters: ['/opt/conda/bin/nnUNet_predict', '-i', '/workdir/bibsnet/sub-BB03601/ses-1/input', '-o', '/workdir/bibsnet/sub-BB03601/ses-1/output', '-t', '540', '-m', '3d_fullres', '--disable_tta'] ERROR 2024-09-13 12:56:25,103: nnUNet failed to complete, exitcode -9 Error: Output segmentation file not created at the path below during nnUNet_predict run. /workdir/bibsnet/sub-BB03601/ses-1/output For your input files at the path below, check their filenames and visually inspect them if needed. /workdir/bibsnet/sub-BB03601/ses-1/input ```

The centering looks better now. T1_2 T2_2

scott-huberty commented 2 months ago

@dnkennedy would you mind editing your message (ellipsis in the top right) and wrapping all that BIBSnet output in triple backticks? i.e. like

``` WARNING: underlay of /etc/localtime required more than 50 (113) bind mounts WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (496) bind mounts

...

For your input files at the path below, check their filenames and visually inspect them if needed. /workdir/bibsnet/sub-BB03601/ses-1/input ```

Which will make it easier to read 🙂 like below:

WARNING: underlay of /etc/localtime required more than 50 (113) bind mounts
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (496) bind mounts

...

For your input files at the path below, check their filenames and visually inspect them if needed.
/workdir/bibsnet/sub-BB03601/ses-1/input
dnkennedy commented 2 months ago

@scott-huberty I tried the triple tics AND a trick Yarik taught me, to make it collapsible... Let me know if that is too much.

scott-huberty commented 2 months ago

@scott-huberty I tried the triple tics AND a trick Yarik taught me, to make it collapsible... Let me know if that is too much.

Nope I Love the Details element 🙂

LuciMoore commented 2 months ago

@dnkennedy It looks like the overcropping was resolved, so that's good. Unfortunately nnUNet_predict is a bit of a black box to me without more info (unless this is another error caused by the configuration of the container). Perhaps it's running out of memory? What kind of computing resources are you using? We recently updated the documentation to specify:

When running BIBSnet using a GPU, the job typically requires about 45 minutes, 20 tasks, and one node with 40 GB of memory. However, we have also had success running BIBSNet on a CPU with 40 GB of RAM.

Let me know if you're able to share the input/output data or some part of it. I'll see if @tjhendrickson and @paul-reiners (who developed/trained the nnUNet model used by BIBSNet) have thoughts on other mechanisms for troubleshooting

dnkennedy commented 2 months ago

Thanks, @LuciMoore . I'm running using GPU on my university cluster. I'll double check the memory allocation and make sure it's 40GB.

scott-huberty commented 2 months ago

@dnkennedy what I usually do at this point is run BIBsnet with some test file to determine if the issue is with my specific file or with the image/system etc. Sorry for the long code but this is what I did when I was having issues with BIBSnet:

# This assumes you are NOT using a windows OS.

# Create a virtual environment in the 'my_env' directory (if you don't already have a virtual environment)
python3 -m venv my_env

# Activate the virtual environment (Unix-based systems)
source my_env/bin/activate

# Install openneuro within the virtual environment
python3 -m pip install openneuro-py

# Start a python session
python3
# Python

from pathlib import Path
import openneuro

# Create the directory for the dataset
Path("./ds004776").mkdir()

# Download the dataset to the d004776 directory, using openneuro
openneuro.download(dataset="ds004776", target_dir="./ds004776", include="sub-01")

# Close the python session
exit()

Run your singularity command as usual, adjusting the paths to this new test file


singularity run --nv --cleanenv --no-home \
    -B /path/to/ds004776:/input \
    -B /path/to/derivatives:/output \
    /path/to/bibsnet.sif \
    /input /output participant \
    -participant 01

If you do this, let us know if the pipeline successfully runs with this file!

LuciMoore commented 2 months ago

@dnkennedy can you try running with -d instead of -v? hopefully that will give more useful information from nnUNet!

dnkennedy commented 2 months ago

@scott-huberty @LuciMoore the test worked beautifully... test_screenshot

will work on the sharing of my example data...

my data is from Philips (should have mentioned earlier), and was BIDSified via heudiconv. I have a couple (well 3 each) of T1s and T2s, for what thats worth.

tree . └── ses-1 ├── anat │   ├── sub-BB036_ses-1_acq-MRPAGE_run-10_T1w.json │   ├── sub-BB036_ses-1_acq-MRPAGE_run-10_T1w.nii.gz │   ├── sub-BB036_ses-1_acq-MRPAGE_run-14_T1w.json │   ├── sub-BB036_ses-1_acq-MRPAGE_run-14_T1w.nii.gz │   ├── sub-BB036_ses-1_acq-MRPAGE_run-21_T1w.json │   ├── sub-BB036_ses-1_acq-MRPAGE_run-21_T1w.nii.gz │   ├── sub-BB036_ses-1_run-11_T2w.json │   ├── sub-BB036_ses-1_run-11_T2w.nii.gz │   ├── sub-BB036_ses-1_run-12_T2w.json │   ├── sub-BB036_ses-1_run-12_T2w.nii.gz │   ├── sub-BB036_ses-1_run-13_T2w.json │   ├── sub-BB036_ses-1_run-13_T2w.nii.gz │   ├── sub-BB036_ses-1_run-20_T2w.json │   └── sub-BB036_ses-1_run-20_T2w.nii.gz

LuciMoore commented 2 months ago

@dnkennedy awesome! Ok so yes, this must have something to do with the data. are there any notable differences in the metadata for the input files created from your data vs the test data (especially orientation and coordination system info)? was the test data converted with heudiconv as well?

scott-huberty commented 2 months ago

was the test data converted with heudiconv as well?

@LuciMoore I think the data are already BIDS compliant. It's just the infant freesurfer test subject that is on OpenNeuro.

dnkennedy commented 2 months ago

@LuciMoore @scott-huberty some progress. Prompted by the sample data success, I ripped down my data, which had 3 T1s and 4 T2s (or so) to just 1 of each, and it ran ok, I guess.

So I'm now restoring the multiple T1s and T2s to see if that's what introduced the problem. Will report back if that fails with the "-d" output.

1anat_v2

dnkennedy commented 2 months ago

Well, I can not break it any more; after the successful test run, as describer by @scott-huberty above, my runs on a single T1/T2 and multi T1s/T2s; and in the context of my complete study are all operating as expected. I guess the only thing I'm doing a little differently, since I am using a 'working directory' is making sure I'm using an fresh empty directory, and not 'contaminated' with any prior (failed) runs.

So, I guess we can close this, and I'll try to break it some other ways...

scott-huberty commented 2 months ago

Awesome @dnkennedy

If I had a nickel for every time that something magically started working again.. Glad things are working for now!

LuciMoore commented 2 months ago

@dnkennedy Glad to hear it’s running now! Up to this point our team has largely focused on the model training and refinement of preprocessing steps, but the code base itself would certainly benefit from some polish, so keep breaking away! I’ve linked this to an existing issue for our future reference