MICA-MNI / micapipe

micapipe from the Multimodal imaging and connectome analysis lab (http://mica-mni.github.io) at the Montreal Neurological Institute. Read The Docs documentation below
http://micapipe.readthedocs.io
GNU General Public License v3.0
80 stars 27 forks source link

Issue in set-up #91

Closed marcellamontagnese closed 9 months ago

marcellamontagnese commented 11 months ago

Hi, Thanks so much for developing such a useful multimodal pipeline!

I just started using this and I am trying to run the following command from the HPC: singularity run --cleanenv /home/mm2075/rds/hpc-work/Pipelines/micapipe/micalab_micapipe_v0.2.2-2023-08-04-c796d70fe910.simg \ -bids ${bids_dir} \ -sub ${subj} \ -out ${out_dir} \ -ses ${session} \ -tmpDir ${temp_dir}/work_dir/sub-${subj}_ses-${session} \ -fs_licence /home/mm2075/rds/hpc-work/Pipelines/license.txt \ -proc_structural

However the pipeline stops within a few seconds with error: [ ERROR ]..... sub-33909_ses-20220505_space-nativepro_T1w was not generated It seems to not be able to locate files correctly and fails at several steps based on the output below:

mm2075 @ login-e-2 Wed Aug 9 16:46:24 BST 2023: COMMAND --> fslreorient2std /rds/user/mm2075/hpc-work/Pipelines/micapipe/work_dir/sub-33909_ses-20220505/21176_micapipe_proc_struc-vol_33909/obl-33909_ses-20220505_T1w.nii.gz /rds/user/mm2075/hpc-work/Pipelines/micapipe/work_dir/sub-33909_ses-20220505/21176_micapipe_proc_struc-vol_33909/reo-33909_ses-20220505_T1w.nii.gz

Traceback (most recent call last): File "/opt/fsl-6.0.2/bin/imcp", line 6, in <module> from fsl.scripts.imcp import main File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/fsl/scripts/imcp.py", line 26, in <module> import fsl.utils.imcp as imcp File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/fsl/utils/imcp.py", line 21, in <module> import nibabel as nib File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/__init__.py", line 38, in <module> from . import analyze as ana File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/analyze.py", line 87, in <module> from .volumeutils import (native_code, swapped_code, make_dt_codes, File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/volumeutils.py", line 22, in <module> from .casting import (shared_range, type_info, OK_FLOATS) File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/casting.py", line 11, in <module> from .testing import setup_test # noqa File "/opt/fsl-6.0.2/fslpython/envs/fslpython/lib/python3.7/site-packages/nibabel/testing/__init__.py", line 21, in <module> from numpy.testing.decorators import skipif ModuleNotFoundError: No module named 'numpy.testing.decorators'

Thanks a lot for the help!

rcruces commented 11 months ago

Hello, maybe the issue is because directories that are not correctly being mounted inside the singularity container. Perhaps you can try to use the option -B

  -B, --bind strings                  a user-bind path specification. 
                                      spec has the format
                                      src[:dest[:opts]], where src and
                                      dest are outside and inside paths. 
                                      If dest is not given, it is set
                                      equal to src.  Mount options
                                      ('opts') may be specified as 'ro'
                                      (read-only) or 'rw' (read/write,
                                      which is the default). Multiple bind
                                      paths can be given by a comma
                                      separated list.

Like here:

micapipe_image=/home/mm2075/rds/hpc-work/Pipelines/micapipe/micalab_micapipe_v0.2.2-2023-08-04-c796d70fe910.simg

singularity run --cleanenv \
    -B ${bids_dir}:/bids\
    -B ${out_dir}:/out \
    -B ${temp_dir}/work_dir/sub-${subj}_ses-${session}:/tmp \
    -B /home/mm2075/rds/hpc-work/Pipelines/license.txt:/opt/licence.txt \
    ${micapipe_image} -bids /bids -out /out -tmpDir /tmp -fs_licence /opt/licence.txt \
     -sub ${subj} -ses ${session} \ 
     -proc_structural

You can also increase the number of threads with -threads for faster computation, the default is 6. Let me know if it works. Also could you download the latest dockerhub container, we had a typo in one module and we replace the image yesterday.

Additionally what is the log of this COMMAND --> 3dresample? which is previous of fslreorient2std. It looks like there is something missing on fslpython. I just ran two test on two different datasets and I could not replicated that python error. ModuleNotFoundError: No module named 'numpy.testing.decorators'

marcellamontagnese commented 11 months ago

Thanks a lot for the help @rcruces That definitely fixed some of the earlier errors!

Now I just get the problem with numpy.testing.decorators The only way I can get things to run is if I bind it to an older FSL version like singularity run --cleanenv -B /usr/local/Cluster-Apps/fsl/5.0.8/fsl:/opt/fsl-6.0.2 ${micapipe_image} \ -bids ${bids_dir} \ -sub ${subj} \ -out ${out_dir} \ -ses ${session} \ -tmpDir ${temp_dir}/work_dir/sub-${subj}_ses-${session} \ -fs_licence /home/mm2075/rds/hpc-work/Pipelines/license.txt \ -proc_structural

Could this be related to this issue? https://www.nemotos.net/?p=3643

Thanks again for the assistance

rcruces commented 11 months ago

The container has it's own fsl version that should work and be compatible with all the other software, as well as it's own working conda environment. it does not need external versions of fsl. @PeerHerholz What do you think?

This are the default software dependencies of the pipeline within the container:

[ INFO ]..... MICA pipe - Software versions 
        MRtrix3.....    3.0.1
                        /opt/miniconda-22.11.1/envs/micapipe/bin/mrinfo
        FSL.........    6.0
                        /opt/fsl-6.0.2
        ANFI........     Jun 23 2023 (Version AFNI_23.1.09 'Publius Helvius Pertinax')
                        /opt/afni-latest/3dresample
        ANTS........     2.3.3.dev168-g29bdf
                        /opt/ants-2.3.4/
        WorkBench...    1.3.2
                        /usr/bin/wb_command
        FreeSurfer..    freesurfer-linux-ubuntu18_x86_64-7.3.2-20220804-6354275
                        /opt/freesurfer-7.3.2
        fix.........    /opt/fix1.068/fix
                        /opt/fix1.068
        python......    3.9.16
                        /opt/miniconda-22.11.1/envs/micapipe/bin/python
        R...........    3.6.3
                        /usr/bin/R
        conda.......    conda 22.11.1
                        /opt/miniconda-22.11.1/condabin/conda
araikes commented 11 months ago

Try the --containall flag instead of --cleanenv. You'll have to explicitly bind all of the folders you need (e.g. your bids folder) but it ensures that nothing from your home folder is sneaking into the paths. If your python packages are stored in your home folder, then it's potentially going to grab a wrong numpy version.

marcellamontagnese commented 11 months ago

Thanks a lot both for the very quick and helpful replies! I tried --containall and it worked with the explict bind

Now the structural pipelines seems to run fine until it finishes and says status incomplete. I looked at the log and seems to be an issue at the mrmath step where it says : mrmath: [SYSTEM FATAL CODE: SIGBUS (7)] Bus error: Accessing invalid address (out of storage space?) mrcalc: [01;31m[ERROR] Could not interpret string "-" as either an image path or a numerical value[0m mrcalc: [01;31m[ERROR] As image: [0m mrcalc: [01;31m[ERROR] no filename supplied to standard input (broken pipe?)[0m mrcalc: [01;31m[ERROR] error opening image "-"[0m mrcalc: [01;31m[ERROR] As numerical value: [0m mrcalc: [01;31m[ERROR] error converting string "-" to complex float (no valid conversion)[0m [38;5;75m

I definitely have enough space on the temporary working directory. I am attaching the log file in case it is helpful. proc_structural_2023-08-12_12.11.55.txt

Thanks again for the assistance!

rcruces commented 10 months ago

It looks like this issue is due to storage on your temporary directory, maybe this could help you: https://community.mrtrix.org/t/out-of-storage-space/2792 https://community.mrtrix.org/t/dwi2response-bus-error/236/15

That particular pipeline might be writing files in a directory

MRTRIX_TMPFILE_DIR

marcellamontagnese commented 10 months ago

I managed to fix the issue following your helpful suggestion!

I have so far been able to run the processing for the T1w and FLAIR + post processing, surf and GD - but I am encountering issues with the diffusion and resting state processing.

DWI processing

To account for the AP and PA data I tried to run the following: -proc_dwi \ -dwi_main ${bids_dir}/sub-${subj}/ses-${session}/dwi/sub-${subj}_ses-${session}_dir-AP_dwi.nii.gz \ -dwi_rpe ${bids_dir}/sub-${subj}/ses-${session}/dwi/sub-${subj}_ses-${session}_dir-PA_dwi.nii.gz \ -rpe_all

It all starts ok but then I get an error saying dwifslpreproc: [ERROR] Unable to determine matching reversed phase-encode direction volume for DWI volume 0 See attached log for further info slurm-25573256.txt I wonder if it is related to this I found online or something else I might be setting wrong - https://community.mrtrix.org/t/dwipreproc-rpe-all/2978/10 and https://mrtrix.readthedocs.io/en/dev/dwi_preprocessing/dwifslpreproc.html

func processing

Traceback (most recent call last): File "/opt/freesurfer-7.3.2/python/scripts/mri_synthseg", line 22, in <module> import tensorflow as tf File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/__init__.py", line 41, in <module> from tensorflow.python.tools import module_util as _module_util File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/__init__.py", line 106, in <module> from tensorflow.python.platform import test File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/platform/test.py", line 24, in <module> from tensorflow.python.framework import test_util as _test_util File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/tensorflow/python/framework/test_util.py", line 37, in <module> from absl.testing import parameterized File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/parameterized.py", line 218, in <module> from absl.testing import absltest File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/absltest.py", line 242, in <module> get_default_test_tmpdir(), File "/opt/freesurfer-7.3.2/python/lib/python3.8/site-packages/absl/testing/absltest.py", line 180, in get_default_test_tmpdir tmpdir = os.path.join(tempfile.gettempdir(), 'absl_testing') File "/opt/freesurfer-7.3.2/python/lib/python3.8/tempfile.py", line 286, in gettempdir tempdir = _get_default_tempdir() File "/opt/freesurfer-7.3.2/python/lib/python3.8/tempfile.py", line 218, in _get_default_tempdir raise FileNotFoundError(_errno.ENOENT, FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/home/mica']

Also attaching the log in case it is useful slurm-25573523.txt

Thanks a lot for your time and assistance!

rcruces commented 10 months ago

Hello,

for the -proc_dwi issue if you use the flag -rpe_all is because when each of the directions of the PA file have one AP and the bvals and bvec files of both PA and AP match. If this is the case of your data, maybe the threshold for the b0 is too high of too low. you can use the threshold flag for identifying the b0 -b0thr.

micapipe -b0thr : Specifies the b-value threshold for determining those image volumes that correspond to b=0. Default = 61 . This value will vary from DWI acquisition and from scanner to scanner (3T or 7T)

This is the function that is being called: dwifslpreproc -rpe_all Specify that ALL DWIs have been acquired with opposing phase-encoding.

If you don't use the flag -rpe_all the correction will be perform based on the b0 volumes only, and it's pretty good as well.

proc_func

Here it looks like you have an error mounting/reading the temporary directory. Check that the directory have read, write and execute permissions, enough space. And maybe unable your local environment variables before running the pipeline like:

unset TMPDIR
unset TEMP

also mount the directory with -B <path_to_tmp>:/tmp. I hope this is helpful

marcellamontagnese commented 10 months ago

Thanks a lot - adding those edits seems to have fixed the issue. The only thing not working now for the functional processing seems to be an issue with FIX where I get:

COMMAND --> [ fix /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC /opt/micapipe/functions/MICAMTL_training_15HC_15PX.RData 20 -m -h 100 FIX Feature extraction for Melodic output directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC create edge masks run FAST registration of standard space masks extract features FIX Classifying components in Melodic directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC using training file: /opt/micapipe/functions/MICAMTL_training_15HC_15PX.RData and threshold 20 FIX features not already generated, so creating them now. FIX Feature extraction for Melodic output directory: /home/mm2075/rds/rds-cam-clin-qmin-nCu8GHKxWRw/data/micapipe/work_dir/2028_micapipe_proc-func_sub-35609_ses-20230411/ICA_MELODIC create edge masks run FAST registration of standard space masks extract features No valid labelling file specified log_func.txt

when then looking at the fix folder in the tmp directory, I found the following error log called logMatlab.txt Setting up environment variables LD_LIBRARY_PATH is .:/opt/matlabmcr-2017b/v93/runtime/glnxa64:/opt/matlabmcr-2017b/v93/bin/glnxa64:/opt/matlabmcr-2017b/v93/sys/os/glnxa64:/opt/matlabmcr-2017b/v93/sys/opengl/lib/glnxa64 Could not access the MATLAB Runtime component cache. Details: Error in initializing CTF core properties. Details: 'MalformedDocumentError:xsd_binder: Attempted to parse a malformed document'

I tried adding set or export MCR_CACHE_ROOT="/data/micapipe/work_dir" and the bind in the submission script but the error persisted

Is this something you have come across before?

Thanks again for all your continued help.

rcruces commented 10 months ago

Hello,

Sorry for the delayed response the fix training file MICAMTL_training_15HC_15PX.RData is specific to the MICs database https://portal.conp.ca/dataset?id=projects/mica-mics. If you want to use this feature you should provide another training file of your own datase https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIX/UserGuide#Training_dataset. Another option would be to use the option -noFIX to skip this step and -NSR for nuisance regression. Spike regression is applied by default.

I hope this is helpful

rcruces commented 9 months ago

I will close this issue for now. If you have further comments regarding this issue please feel free to re-open it. If you have further issues about something else open a new issue.

best.