Closed marcellamontagnese closed 9 months ago
Hello, maybe the issue is because directories that are not correctly being mounted inside the singularity container. Perhaps you can try to use the option -B
-B, --bind strings a user-bind path specification.
spec has the format
src[:dest[:opts]], where src and
dest are outside and inside paths.
If dest is not given, it is set
equal to src. Mount options
('opts') may be specified as 'ro'
(read-only) or 'rw' (read/write,
which is the default). Multiple bind
paths can be given by a comma
separated list.
Like here:
micapipe_image=/home/mm2075/rds/hpc-work/Pipelines/micapipe/micalab_micapipe_v0.2.2-2023-08-04-c796d70fe910.simg
singularity run --cleanenv \
-B ${bids_dir}:/bids\
-B ${out_dir}:/out \
-B ${temp_dir}/work_dir/sub-${subj}_ses-${session}:/tmp \
-B /home/mm2075/rds/hpc-work/Pipelines/license.txt:/opt/licence.txt \
${micapipe_image} -bids /bids -out /out -tmpDir /tmp -fs_licence /opt/licence.txt \
-sub ${subj} -ses ${session} \
-proc_structural
You can also increase the number of threads with -threads
for faster computation, the default is 6. Let me know if it works. Also could you download the latest dockerhub container, we had a typo in one module and we replace the image yesterday.
Additionally what is the log of this COMMAND --> 3dresample
? which is previous of fslreorient2std
. It looks like there is something missing on fslpython
. I just ran two test on two different datasets and I could not replicated that python error.
ModuleNotFoundError: No module named 'numpy.testing.decorators'
Thanks a lot for the help @rcruces That definitely fixed some of the earlier errors!
Now I just get the problem with numpy.testing.decorators
The only way I can get things to run is if I bind it to an older FSL version like singularity run --cleanenv -B /usr/local/Cluster-Apps/fsl/5.0.8/fsl:/opt/fsl-6.0.2 ${micapipe_image} \ -bids ${bids_dir} \ -sub ${subj} \ -out ${out_dir} \ -ses ${session} \ -tmpDir ${temp_dir}/work_dir/sub-${subj}_ses-${session} \ -fs_licence /home/mm2075/rds/hpc-work/Pipelines/license.txt \ -proc_structural
Could this be related to this issue? https://www.nemotos.net/?p=3643
Thanks again for the assistance
The container has it's own fsl version that should work and be compatible with all the other software, as well as it's own working conda environment. it does not need external versions of fsl
. @PeerHerholz What do you think?
This are the default software dependencies of the pipeline within the container:
[ INFO ]..... MICA pipe - Software versions
MRtrix3..... 3.0.1
/opt/miniconda-22.11.1/envs/micapipe/bin/mrinfo
FSL......... 6.0
/opt/fsl-6.0.2
ANFI........ Jun 23 2023 (Version AFNI_23.1.09 'Publius Helvius Pertinax')
/opt/afni-latest/3dresample
ANTS........ 2.3.3.dev168-g29bdf
/opt/ants-2.3.4/
WorkBench... 1.3.2
/usr/bin/wb_command
FreeSurfer.. freesurfer-linux-ubuntu18_x86_64-7.3.2-20220804-6354275
/opt/freesurfer-7.3.2
fix......... /opt/fix1.068/fix
/opt/fix1.068
python...... 3.9.16
/opt/miniconda-22.11.1/envs/micapipe/bin/python
R........... 3.6.3
/usr/bin/R
conda....... conda 22.11.1
/opt/miniconda-22.11.1/condabin/conda
Try the --containall
flag instead of --cleanenv
. You'll have to explicitly bind all of the folders you need (e.g. your bids folder) but it ensures that nothing from your home folder is sneaking into the paths. If your python packages are stored in your home folder, then it's potentially going to grab a wrong numpy
version.
Thanks a lot both for the very quick and helpful replies!
I tried --containall
and it worked with the explict bind
Now the structural pipelines seems to run fine until it finishes and says status incomplete. I looked at the log and seems to be an issue at the mrmath step where it says : mrmath: [SYSTEM FATAL CODE: SIGBUS (7)] Bus error: Accessing invalid address (out of storage space?) mrcalc: [01;31m[ERROR] Could not interpret string "-" as either an image path or a numerical value[0m mrcalc: [01;31m[ERROR] As image: [0m mrcalc: [01;31m[ERROR] no filename supplied to standard input (broken pipe?)[0m mrcalc: [01;31m[ERROR] error opening image "-"[0m mrcalc: [01;31m[ERROR] As numerical value: [0m mrcalc: [01;31m[ERROR] error converting string "-" to complex float (no valid conversion)[0m [38;5;75m
I definitely have enough space on the temporary working directory. I am attaching the log file in case it is helpful. proc_structural_2023-08-12_12.11.55.txt
Thanks again for the assistance!
It looks like this issue is due to storage on your temporary directory, maybe this could help you: https://community.mrtrix.org/t/out-of-storage-space/2792 https://community.mrtrix.org/t/dwi2response-bus-error/236/15
That particular pipeline might be writing files in a directory
MRTRIX_TMPFILE_DIR
I managed to fix the issue following your helpful suggestion!
I have so far been able to run the processing for the T1w and FLAIR + post processing, surf and GD - but I am encountering issues with the diffusion and resting state processing.
dwi
- I have reversed phase encoding (AP and PA) for all DWIs data (all with the same number of volumes, bvals, bvecs). My BIDS folder contains the following files:
├── sub-35843_ses-20230327_dir-AP_dwi.bval
├── sub-35843_ses-20230327_dir-AP_dwi.bvec
├── sub-35843_ses-20230327_dir-AP_dwi.json
├── sub-35843_ses-20230327_dir-AP_dwi.nii.gz
├── sub-35843_ses-20230327_dir-PA_dwi.bval
├── sub-35843_ses-20230327_dir-PA_dwi.bvec
├── sub-35843_ses-20230327_dir-PA_dwi.json
└── sub-35843_ses-20230327_dir-PA_dwi.nii.gz
To account for the AP and PA data I tried to run the following:
-proc_dwi \ -dwi_main ${bids_dir}/sub-${subj}/ses-${session}/dwi/sub-${subj}_ses-${session}_dir-AP_dwi.nii.gz \ -dwi_rpe ${bids_dir}/sub-${subj}/ses-${session}/dwi/sub-${subj}_ses-${session}_dir-PA_dwi.nii.gz \ -rpe_all
It all starts ok but then I get an error saying dwifslpreproc: [ERROR] Unable to determine matching reversed phase-encode direction volume for DWI volume 0
See attached log for further info
slurm-25573256.txt
I wonder if it is related to this I found online or something else I might be setting wrong - https://community.mrtrix.org/t/dwipreproc-rpe-all/2978/10 and https://mrtrix.readthedocs.io/en/dev/dwi_preprocessing/dwifslpreproc.html
func
processing I keep getting a series of related errors to do with Freesurfer saying something like this:Traceback (most recent call last): File "/opt/freesurfer-7.3.2/python/scripts/mri_synthseg", line 22, in <module> import tensorflow as tf File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/__init__.py", line 41, in <module> from tensorflow.python.tools import module_util as _module_util File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/__init__.py", line 106, in <module> from tensorflow.python.platform import test File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/platform/test.py", line 24, in <module> from tensorflow.python.framework import test_util as _test_util File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/framework/test_util.py", line 37, in <module> from absl.testing import parameterized File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/parameterized.py", line 218, in <module> from absl.testing import absltest File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/absltest.py", line 242, in <module> get_default_test_tmpdir(), File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/absltest.py", line 180, in get_default_test_tmpdir tmpdir = os.path.join(tempfile.gettempdir(), 'absl_testing') File "/opt/freesurfer-7.3.2/python/lib/python3.8/tempfile.py", line 286, in gettempdir tempdir = _get_default_tempdir() File "/opt/freesurfer-7.3.2/python/lib/python3.8/tempfile.py", line 218, in _get_default_tempdir raise FileNotFoundError(_errno.ENOENT, FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/home/mica']
Also attaching the log in case it is useful slurm-25573523.txt
Thanks a lot for your time and assistance!
Hello,
for the -proc_dwi
issue if you use the flag -rpe_all
is because when each of the directions of the PA file have one AP and the bvals and bvec files of both PA and AP match. If this is the case of your data, maybe the threshold for the b0 is too high of too low. you can use the threshold flag for identifying the b0 -b0thr
.
micapipe
-b0thr : Specifies the b-value threshold for determining those image volumes that correspond to b=0.
Default = 61 . This value will vary from DWI acquisition and from scanner to scanner (3T or 7T)
This is the function that is being called:
dwifslpreproc
-rpe_all
Specify that ALL DWIs have been acquired with opposing phase-encoding.
If you don't use the flag -rpe_all
the correction will be perform based on the b0 volumes only, and it's pretty good as well.
Here it looks like you have an error mounting/reading the temporary directory. Check that the directory have read, write and execute permissions, enough space. And maybe unable your local environment variables before running the pipeline like:
unset TMPDIR
unset TEMP
also mount the directory with -B <path_to_tmp>:/tmp
. I hope this is helpful
Thanks a lot - adding those edits seems to have fixed the issue. The only thing not working now for the functional processing seems to be an issue with FIX where I get:
COMMAND --> [ fix /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC /opt/micapipe/functions/MICAMTL_training_15HC_15PX.RData 20 -m -h 100 FIX Feature extraction for Melodic output directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC create edge masks run FAST registration of standard space masks extract features FIX Classifying components in Melodic directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC using training file: /opt/micapipe/functions/MICAMTL_training_15HC_15PX.RData and threshold 20 FIX features not already generated, so creating them now. FIX Feature extraction for Melodic output directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC create edge masks run FAST registration of standard space masks extract features No valid labelling file specified log_func.txt
when then looking at the fix folder in the tmp directory, I found the following error log called logMatlab.txt
Setting up environment variables LD_LIBRARY_PATH is .:/opt/matlabmcr-2017b/v93/runtime/glnxa64:/opt/matlabmcr-2017b/v93/bin/glnxa64:/opt/matlabmcr-2017b/v93/sys/os/glnxa64:/opt/matlabmcr-2017b/v93/sys/opengl/lib/glnxa64 Could not access the MATLAB Runtime component cache. Details: Error in initializing CTF core properties. Details: 'MalformedDocumentError:xsd_binder: Attempted to parse a malformed document'
I tried adding set
or export MCR_CACHE_ROOT="/data/micapipe/work_dir"
and the bind in the submission script but the error persisted
Is this something you have come across before?
Thanks again for all your continued help.
Hello,
Sorry for the delayed response the fix training file MICAMTL_training_15HC_15PX.RData
is specific to the MICs database https://portal.conp.ca/dataset?id=projects/mica-mics. If you want to use this feature you should provide another training file of your own datase https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIX/UserGuide#Training_dataset. Another option would be to use the option -noFIX
to skip this step and -NSR
for nuisance regression. Spike regression is applied by default.
I hope this is helpful
I will close this issue for now. If you have further comments regarding this issue please feel free to re-open it. If you have further issues about something else open a new issue.
best.
Hi, Thanks so much for developing such a useful multimodal pipeline!
I just started using this and I am trying to run the following command from the HPC:
singularity run --cleanenv /home/mm2075/rds/hpc-work/Pipelines/micapipe/micalab_micapipe_v0.2.2-2023-08-04-c796d70fe910.simg \ -bids ${bids_dir} \ -sub ${subj} \ -out ${out_dir} \ -ses ${session} \ -tmpDir ${temp_dir}/work_dir/sub-${subj}_ses-${session} \ -fs_licence /home/mm2075/rds/hpc-work/Pipelines/license.txt \ -proc_structural
However the pipeline stops within a few seconds with error:
[ ERROR ]..... sub-33909_ses-20220505_space-nativepro_T1w was not generated
It seems to not be able to locate files correctly and fails at several steps based on the output below:mm2075 @ login-e-2 Wed Aug 9 16:46:24 BST 2023: COMMAND --> fslreorient2std /rds/user/mm2075/hpc-work/Pipelines/micapipe/work_dir/sub-33909_ses-20220505/21176_micapipe_proc_struc-vol_33909/obl-33909_ses-20220505_T1w.nii.gz /rds/user/mm2075/hpc-work/Pipelines/micapipe/work_dir/sub-33909_ses-20220505/21176_micapipe_proc_struc-vol_33909/reo-33909_ses-20220505_T1w.nii.gz
Traceback (most recent call last): File "/opt/fsl-6.0.2/bin/imcp", line 6, in <module> from fsl.scripts.imcp import main File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/fsl/scripts/imcp.py", line 26, in <module> import fsl.utils.imcp as imcp File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/fsl/utils/imcp.py", line 21, in <module> import nibabel as nib File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/__init__.py", line 38, in <module> from . import analyze as ana File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/analyze.py", line 87, in <module> from .volumeutils import (native_code, swapped_code, make_dt_codes, File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/volumeutils.py", line 22, in <module> from .casting import (shared_range, type_info, OK_FLOATS) File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/casting.py", line 11, in <module> from .testing import setup_test # noqa File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/testing/__init__.py", line 21, in <module> from numpy.testing.decorators import skipif ModuleNotFoundError: No module named 'numpy.testing.decorators'
Thanks a lot for the help!