FastSurfer tries to run singularity w/in containerized call and fails

bcmcpher commented 11 months ago

In attempting to run the latest version (v0.2.2) on a UKBB subject I get an early failure during surface creation.

The call made:

apptainer run --cleanenv \
    -B ${WORKDIR}/bids:/bids \
    -B ${WORKDIR}/derivatives:/out \
    -B ${WORKDIR}/work:/tmp \
    -B ${WORKDIR}/bin/license.txt:/opt/licence.txt \
    ${WORKDIR}/container/micapipe-v0.2.2.simg \
    -bids /bids \
    -sub ${SUBJ} \
    -ses 2 \
    -out /out \
    -fs_licence /opt/licence.txt \
    -proc_structural -proc_surf -post_structural -GD \
    -proc_flair -regSynth \
    -MPC -microstructural_img ${BIDSDIR}/swi/${SUBJ}_${SESS}_swi.nii.gz -microstructural_reg FALSE -mpc_acq SWI \
    -proc_dwi -dwi_main ${BIDSDIR}/dwi/${SUBJ}_${SESS}_acq-AP_dwi.nii.gz -dwi_rpe ${BIDSDIR}/dwi/${SUBJ}_${SESS}_acq-PA_dwi.nii.gz \
    -SC -tracts 2M -keep_tck -autoTract \
    -proc_func -mainScanStr task-rest_bold -GSR \
    -QC_subj -threads 6

Based on a quick pass, it looks like a version check fails over to running FastSurfer as a singularity call instead of bare metal in the container. The first error is singularity not found and the following errors are output/file not found.
The output directory is still saying v0.2.0, could this be breaking a path / file exists check?
This is run on Compute Canada (Beluga), I can verify any additional details if they're relevant.

This is where the logic kicks over to the singularity call: https://github.com/MICA-MNI/micapipe/blob/master/functions/01_proc-surf.sh#L133C27-L158

The logs/procsurf*.txt says:

        Surface software    fastsurfer

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:34:17 UTC 2023:
COMMAND -->   mkdir -p /out/fastsurfer/fsaverage5/surf  

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:34:18 UTC 2023:
COMMAND -->   mkdir -p /out/fastsurfer/fsLR-32k/surf  

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:34:20 UTC 2023:
COMMAND -->   mkdir -p /out/fastsurfer/fsLR-5k/surf  

-------------------------------------------------------------
    Surface processing
        micapipe v0.2.2 'Northern flicker', container_micapipe-v0.2.2 
-------------------------------------------------------------

[ INFO ]..... MICA pipe - Software versions 
        MRtrix3.....    3.0.1
                        /opt/miniconda-22.11.1/envs/micapipe/bin/mrinfo
        FSL.........    6.0
                        /opt/fsl-6.0.2
        ANFI........     Jun 23 2023 (Version AFNI_23.1.09 'Publius Helvius Pertinax')
                        /opt/afni-latest/3dresample
        ANTS........     2.3.3.dev168-g29bdf
                        /opt/ants-2.3.4/
        WorkBench...    1.3.2
                        /usr/bin/wb_command
        FreeSurfer..    freesurfer-linux-ubuntu18_x86_64-7.3.2-20220804-6354275
                        /opt/freesurfer-7.3.2
        fix.........    /opt/fix1.068/fix
                        /opt/fix1.068
        python......    3.9.16
                        /opt/miniconda-22.11.1/envs/micapipe/bin/python
        R...........    3.6.3
                        /usr/bin/R
        conda.......    conda 22.11.1
                        /opt/miniconda-22.11.1/condabin/conda

[ INFO ]..... mica-pipe inputs 
        id   :  1000011
        BIDS :  /bids
        out  :  /out/micapipe_v0.2.0
        ses  :  ses-2

[ INFO ]..... BIDS naming 
        subject_bids :  /bids/sub-1000011/ses-2
        bids_T1ws    :  N-1, e.g. /bids/sub-1000011/ses-2/anat/sub-1000011_ses-2_T1w.nii.gz
        bids_dwis    :  N-0, e.g. 
        subject      :  sub-1000011
        subject_dir  :  /out/micapipe_v0.2.0/sub-1000011/ses-2
        proc_struct  :  /out/micapipe_v0.2.0/sub-1000011/ses-2/anat
        dir_warp     :  /out/micapipe_v0.2.0/sub-1000011/ses-2/xfm
        logs         :  /out/micapipe_v0.2.0/sub-1000011/ses-2/logs

[ INFO ]..... Processing directories 
        subject_dir     :   /out/micapipe_v0.2.0/sub-1000011/ses-2
        proc_struct     :   /out/micapipe_v0.2.0/sub-1000011/ses-2/anat
        dir_conte69     :   /out/micapipe_v0.2.0/sub-1000011/ses-2/surf
        dir_volum       :   /out/micapipe_v0.2.0/sub-1000011/ses-2/parc
        dir_warp        :   /out/micapipe_v0.2.0/sub-1000011/ses-2/xfm
        dir_logs        :   /out/micapipe_v0.2.0/sub-1000011/ses-2/logs
        dir_QC          :   /out/micapipe_v0.2.0/sub-1000011/ses-2/QC
        dir_subjsurf  : /out/fastsurfer/sub-1000011_ses-2

[ INFO ]..... Utilities directories 
        scriptDir         : /opt/micapipe/functions
        util_MNIvolumes   : /opt/micapipe/MNI152Volumes
        util_lut          : /opt/micapipe/parcellations/lut
        util_parcelations : /opt/micapipe/parcellations
        util_surface      : /opt/micapipe/surfaces
        util_mics         : /opt/micapipe/MICs60_T1-atlas
        Preprocessed surface directory: FALSE

[ INFO ]..... Processing surfaces 
        Saving temporal dir:    FALSE
        Temporal dir:   /tmp/1000011_micapipe_proc-surf_24199
        Parallel processing:    6 threads
        fs licence: /opt/licence.txt
        fastsurfer img: 
        t1 to process:  /out/micapipe_v0.2.0/sub-1000011/ses-2/anat/sub-1000011_ses-2_space-nativepro_T1w.nii.gz

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:42:04 UTC 2023:
COMMAND -->   mkdir -p /tmp/1000011_micapipe_proc-surf_24199/nii  
        PROC    container_micapipe-v0.2.2

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:42:04 UTC 2023:
COMMAND -->   mkdir -p /out/fastsurfer/sub-1000011_ses-2  

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:42:04 UTC 2023:
COMMAND -->   cp /out/micapipe_v0.2.0/sub-1000011/ses-2/anat/sub-1000011_ses-2_space-nativepro_T1w.nii.gz /tmp/1000011_micapipe_proc-surf_24199/nii/sub-1000011_ses-2_T1w.nii.gz  

[ INFO ]..... FastSurfer: running the singularity image 
/opt/micapipe/functions/01_proc-surf.sh: line 150: singularity: command not found
removed '/tmp/1000011_micapipe_proc-surf_24199/nii/sub-1000011_ses-2_T1w.nii.gz'
removed directory '/tmp/1000011_micapipe_proc-surf_24199/nii'
chmod: cannot access '/tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2': No such file or directory
cp: cannot stat '/tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2/mri/norm.mgz': No such file or directory

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:42:05 UTC 2023:
COMMAND -->   cp /tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2/scripts/recon-all.log /out/micapipe_v0.2.0/sub-1000011/ses-2/logs/recon-all.log  
cp: cannot stat '/tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2/scripts/recon-all.log': No such file or directory

bcmcpher @ bc11347.int.ets1.calculquebec.ca Wed Aug  9 20:42:05 UTC 2023:
COMMAND -->   cp -r /tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2 /out/fastsurfer  
cp: cannot stat '/tmp/1000011_micapipe_proc-surf_24199/sub-1000011_ses-2': No such file or directory
        Check log file: /out/micapipe_v0.2.0/sub-1000011/ses-2/logs/recon-all.log

[ INFO ]..... Creating proc_surf json file 

-------------------------------------------------------------
    proc_surf processing ended in  0.017 minutes :
    logs:
        Steps completed : 00/01
        Status          : INCOMPLETE
        Check logs      : /out/micapipe_v0.2.0/sub-1000011/ses-2/logs/proc_surf_2023-08-09_20.34.16.txt
-------------------------------------------------------------

araikes commented 11 months ago

There was a hotfix for a similar issue the other day... When did you pull the container?

bcmcpher commented 11 months ago

Last Friday (2023-08-04) late afternoon, between ~3-7.

Do I just need to rebuild it again?

rcruces commented 11 months ago

Yes, please it was a typo and an issue with the permissions. Just pull the latest container and it should work. Let us know if it worked

bcmcpher commented 10 months ago

It's resolved - the fix worked, I was on an out of date build of the container. Thanks!

MICA-MNI / micapipe

FastSurfer tries to run singularity w/in containerized call and fails #95