FileNotFoundError during funcMRIQC processing

[suxpert]

What happened?

I was using MRIQC docker images, it used almost all memory resources (even I've limited to 80GB), and then raised error with traceback messages (see log below).

What command did you use?

docker run -it --rm -v bids:/data:ro -v mriqc:/out nipreps/mriqc:24.0.0 --notrack --no-sub --mem 80 /data /out participant

What version of the software are you running?

24.0.0 (reported as 24.1.0.dev0...)

How are you running this software?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

2024-08-16 08:10:25 | ERROR    | concurrent.futures | exception calling callback for <Future at 0x7f679f3fdc50 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 497, in run
    self._get_hashval()
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 548, in _get_hashval
    self._get_inputs()
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 590, in _get_inputs
    outputs = _load_resultfile(results_fname).outputs
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
    result = loadpkl(results_file)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/utils/filemanip.py", line 619, in loadpkl
    raise IOError(error_message)
OSError: Result file /tmp/work/mriqc_wf/funcMRIQC/_in_file_..data..sub-0156..func..sub-0156_task-rest_run-01_bold.nii.gz/mean/result_mean.pklz expected, but does not exist after (5.0) seconds.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 72, in run_node
    result['result'] = node.result
                       ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 223, in result
    return _load_resultfile(
           ^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /tmp/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..data..sub-0156..func..sub-0156_task-rest_run-01_bold.nii.gz/pre_clip/result_pre_clip.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 456, in _async_callback
    result = args.result()
             ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
FileNotFoundError: /tmp/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..data..sub-0156..func..sub-0156_task-rest_run-01_bold.nii.gz/pre_clip/result_pre_clip.pklz

Additional information / screenshots

No response

[oesteban]

There are a few improvements in the current master that may resolve this issue.

This looks like a memory error that killed a process ongoing (likely synthstrip), and when nipype comes back to retrieve the results they are missing.

I'd suggest deleting the work directory and trying with the latest version -- is that possible?

[suxpert]

Where could I fetch the docker image for the lastest/nightly version? Or need I build the docker image myself from this rep?

[suxpert]

And BTW, I have limited the memory resource to be 80GB via --mem 80, and MRIQC may use all my memory resources (128G), or even freeze my computer and then crash. Why?

[suxpert]

@oesteban I've tried the latest version (24.0.1), MRIQC works without error for the same dataset, thanks.