Node Name: fmriprep_22_0_wf.single_subject_050100101B_wf.anat_preproc_wf.t1w_dseg

[Tordelap]

What happened?

Hello,

I was trying to preprocess a first subject within a simple data set that just includes T1 and resting state images. I ran the first subject and it reported an error that I do not quite understand. I have had other problems with the BIDS validation and .json files with this sample, but I think I have solved them. Am I doing something wrong? Can you help me please?

Thank you very much!

What command did you use?

fmriprep-docker --fs-license-file /media/usuari/Datos/freesurfer/license.txt /media/usuari/Datos/MBCT_OCD /media/usuari/Datos/MBCT_OCD/out -w /media/usuari/Datos/trash --participant-label sub-050100101B --use-aroma --output-spaces MNI152NLin2009cAsym

What version of fMRIPrep are you running?

fmriprep-docker-22.0.2

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

File: /out/sub-050100101B/log/20221207-111222_7ce36748-255b-46ee-884c-35367198aa16/crash-20221207-234750-root-t1w_dseg-1d395d9c-ad2c-4fed-a027-0e04cb89d853.txt
Working Directory: /scratch/fmriprep_22_0_wf/single_subject_050100101B_wf/anat_preproc_wf/t1w_dseg
Inputs:

    args:
    bias_iters:
    bias_lowpass:
    environ: {'FSLOUTPUTTYPE': 'NIFTI_GZ'}
    hyper:
    img_type:
    in_files:
    init_seg_smooth:
    init_transform:
    iters_afterbias:
    manual_seg:
    mixel_smooth:
    no_bias: True
    no_pve:
    number_classes:
    other_priors:
    out_basename:
    output_biascorrected:
    output_biasfield:
    output_type: NIFTI_GZ
    probability_maps: True
    segment_iters:
    segments: True
    use_priors:
    verbose:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node t1w_dseg.

Cmdline:
    fast -N -p -g -S 1 /scratch/fmriprep_22_0_wf/single_subject_050100101B_wf/anat_preproc_wf/t1w_dseg/sub-050100101B_T1w_corrected_xform_masked.nii.gz
Stdout:

Stderr:
    Killed
Traceback:
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
        setattr(outputs, key, val)
      File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
        value = super(File, self).validate(objekt, name, value, return_pathlike=True)
      File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
        self.error(objekt, name, str(value))
      File "/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error
        raise TraitError(
    traits.trait_errors.TraitError: The 'tissue_class_map' trait of a FASTOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/fmriprep_22_0_wf/single_subject_050100101B_wf/anat_preproc_wf/t1w_dseg/sub-050100101B_T1w_corrected_xform_masked_seg.nii.gz'  was specified.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 401, in run
        outputs = self.aggregate_outputs(runtime)
      File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
        raise FileNotFoundError(msg)
    FileNotFoundError: No such file or directory '/scratch/fmriprep_22_0_wf/single_subject_050100101B_wf/anat_preproc_wf/t1w_dseg/sub-050100101B_T1w_corrected_xform_masked_seg.nii.gz' for output 'tissue_class_map' of a FAST interface

Additional information / screenshots

No response

[effigies]

The interface says it was killed. How much memory do you have available?

[Tordelap]

I am working on a Ubuntu/Linux docker. I have understood that I cannot set a memory limit in this case.

My system has 32 GB RAM memory and 12th Gen Intel® Core™ i9-12900 × 24

[effigies]

I cannot set a memory limit in this case.

I'm not sure what you mean by that. But you should have enough memory. It could be a fluke, and you can just try rerunning. If you still have your working directory, it will try to pick up where it left off.

[Tordelap]

Thank you Chris,

Sorry, I meant that on Ubuntu, apparently, Docker can use all available host memory (32GB in my case) and I have double check this already. I do not think one single subject is using all that memory, right? I am currently running another subject and I will try to re-run the first one as you recommended.

[Tordelap]

Hi,

After trying to re-run the same subject I've got this error:

fmriprep-docker --fs-license-file /media/usuari/Datos/freesurfer/license.txt /media/usuari/Datos/MBCT_OCD /media/usuari/Datos/MBCT_OCD/out -w /media/usuari/Datos/trash --participant-label sub-050100101B --use-aroma --output-spaces MNI152NLin2009cAsym RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=20.10.17 -it -v /media/usuari/Datos/freesurfer/license.txt:/opt/freesurfer/license.txt:ro -v /media/usuari/Datos/MBCT_OCD:/data:ro -v /media/usuari/Datos/MBCT_OCD/out:/out -v /media/usuari/Datos/trash:/scratch nipreps/fmriprep:22.0.2 /data /out participant --participant-label sub-050100101B --use-aroma -w /scratch --output-spaces MNI152NLin2009cAsym You are using fMRIPrep-22.0.2, and a newer version of fMRIPrep is available: 22.1.0. Please check out our documentation about how and when to upgrade: https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading Traceback (most recent call last): File "/opt/conda/bin/fmriprep", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.9/site-packages/fmriprep/cli/run.py", line 39, in main parse_args() File "/opt/conda/lib/python3.9/site-packages/fmriprep/cli/parser.py", line 764, in parse_args config.from_dict({}) File "/opt/conda/lib/python3.9/site-packages/fmriprep/config.py", line 671, in from_dict execution.load(settings, init=initialize('execution'), ignore=ignore) File "/opt/conda/lib/python3.9/site-packages/fmriprep/config.py", line 232, in load cls.init() File "/opt/conda/lib/python3.9/site-packages/fmriprep/config.py", line 472, in init cls._layout = BIDSLayout( File "/opt/conda/lib/python3.9/site-packages/bids/layout/layout.py", line 152, in init indexer(self) File "/opt/conda/lib/python3.9/site-packages/bids/layout/index.py", line 109, in call self._index_dir(self._layout._root, self._config) File "/opt/conda/lib/python3.9/site-packages/bids/layout/index.py", line 193, in _index_dir self._index_dir(d, list(config), default_action=default) File "/opt/conda/lib/python3.9/site-packages/bids/layout/index.py", line 193, in _index_dir self._index_dir(d, list(config), default_action=default) File "/opt/conda/lib/python3.9/site-packages/bids/layout/index.py", line 193, in _index_dir self._index_dir(d, list(config), default_action=default) [Previous line repeated 2 more times] File "/opt/conda/lib/python3.9/site-packages/bids/layout/index.py", line 163, in _index_dir if config_file.exists(): File "/opt/conda/lib/python3.9/pathlib.py", line 1424, in exists self.stat() File "/opt/conda/lib/python3.9/pathlib.py", line 1232, in stat return self._accessor.stat(self) OSError: [Errno 5] Input/output error: '/data/out/sourcedata/freesurfer/fsaverage/xhemi/layout_config.json' fMRIPrep: Please report errors to https://github.com/nipreps/fmriprep/issues

[effigies]

fmriprep-docker /media/usuari/Datos/MBCT_OCD /media/usuari/Datos/MBCT_OCD/out

You're nesting your outputs in your inputs. That's bound to cause problems.

[Tordelap]

Thank you Chris, I will correct this. Also, my internal disk died and that might be the reason of the memory problems and other errors. Maybe it was a hardware problem after all.

[Tordelap]

Hello Chris,

I have re-run the same participant and it ended successfully, it was indeed a memory problem. I am getting this warning for every subject:

nipype.workflow WARNING: Some nodes exceed the total amount of memory available (27.99GB)

Should I use a --low-mem flag or a --mem-mb 26000 for the remaining sample? What is the best approach? Or should I erase the first participant and run the whole sample with the same flag? Does this flags affect the preprocessing pipeline (any step) or just increase preprocessing times?

Another possible approach, but a less comfortable one, is to re-run every time a participant's process is killed...

Thank you very much in advance,

Víctor.

[effigies]

The --low-mem flag does very little anymore. If you are using ICA-AROMA it will have an impact; otherwise we've been able to reduce the memory consumption of the other nodes that we targeted. The main impact is that some files that would have been compressed will be uncompressed to allow for memory mapping, so the cost is disk space, not time.

--mem-mb will only change how much memory fMRIPrep thinks is available, which won't change some nodes exceeding 28GB.

Another possible approach, but a less comfortable one, is to re-run every time a participant's process is killed...

Agreed that it's not ideal, but memory crashes are hard to eliminate completely for such a large workflow. If you can keep the same working directory around, then it should not be too onerous. You could even try to auto-retry:

for TRY in 1 2 3; do
    fmriprep-docker ... && break
done

[Tordelap]

Thank you very much Chris for your detailed response, I guess I can keep the same working directory and I will try auto-retry!