Exception raised while executing Node synthstrip. (new with 23.0.0 ?)

[yarikoptic]

What happened?

During CI testing of https://github.com/ReproNim/containers/ after some changes (which should not be related but who knows?) and using mriqc 23.0.0 in how we do it the mriqc ran from singularity

started (https://github.com/ReproNim/containers/actions/runs/4506974074/jobs/7934313945> to fail with (click to expand)

```shell [INFO] == Command start (output follows) ===== 230324-02:00:23,760 cli IMPORTANT: Running MRIQC version 23.0.0: * BIDS dataset path: /tmp/repro-VaY1i70/ds000003-qc/sourcedata. * Output folder: /tmp/repro-VaY1i70/ds000003-qc. * Analysis levels: ['participant', 'group']. 230324-02:00:38,526 cli WARNING: IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQMs can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement. 230324-02:05:38,657 nipype.workflow WARNING: [Node] Error on "mriqc_wf.anatMRIQC.synthstrip_wf.synthstrip" (/tmp/repro-VaY1i70/ds000003-qc/workdir/mriqc_wf/anatMRIQC/synthstrip_wf/_in_file_..tmp..repro-VaY1i70..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/synthstrip) 230324-02:05:39,637 nipype.workflow ERROR: Node synthstrip.a1 failed to run on host fv-az1031-255. 230324-02:05:39,646 nipype.workflow ERROR: Saving crash info to /tmp/repro-VaY1i70/ds000003-qc/logs/crash-20230324-020539-runner-synthstrip.a1-93bacc0d-4206-4313-bb94-f258a04c3530.txt Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/mriqc/engine/plugin.py", line 60, in run_node result["result"] = node.run(updatehash=updatehash) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthstrip. Cmdline: synthstrip -b 1 -i /tmp/repro-VaY1i70/ds000003-qc/workdir/mriqc_wf/anatMRIQC/synthstrip_wf/_in_file_..tmp..repro-VaY1i70..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/pre_n4/clipped_corrected.nii.gz --model /opt/freesurfer/models/synthstrip.1.pt -o clipped_corrected_desc-brain.nii.gz -m clipped_corrected_desc-brain_mask.nii.gz Stdout: Stderr: Killed Traceback: RuntimeError: subprocess exited with code 137. Traceback (most recent call last): File "/opt/conda/bin/mriqc", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.9/site-packages/mriqc/cli/run.py", line 167, in main mriqc_wf.run(**_plugin) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run runner.run(execgraph, updatehash=updatehash, config=self.config) File "/opt/conda/lib/python3.9/site-packages/mriqc/engine/plugin.py", line 1[84](https://github.com/ReproNim/containers/actions/runs/4506974074/jobs/7934313945#step:4:85), in run self._clean_queue(jobid, graph, result=result) File "/opt/conda/lib/python3.9/site-packages/mriqc/engine/plugin.py", line 256, in _clean_queue raise RuntimeError("".join(result["traceback"])) RuntimeError: Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/mriqc/engine/plugin.py", line 60, in run_node result["result"] = node.run(updatehash=updatehash) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthstrip. Cmdline: synthstrip -b 1 -i /tmp/repro-VaY1i70/ds000003-qc/workdir/mriqc_wf/anatMRIQC/synthstrip_wf/_in_file_..tmp..repro-VaY1i70..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/pre_n4/clipped_corrected.nii.gz --model /opt/freesurfer/models/synthstrip.1.pt -o clipped_corrected_desc-brain.nii.gz -m clipped_corrected_desc-brain_mask.nii.gz Stdout: Stderr: Killed Traceback: RuntimeError: subprocess exited with code 137. [INFO] == Command exit (modification check follows) ===== ```

Prior successfull run in https://github.com/ReproNim/containers/actions/runs/4496233226/jobs/7910699465 had containers/images/bids/bids-mriqc--0.16.0.sing so I now suspect new version of mriqc to cause this problem

What command did you use?

Such testing run should be redoable on Linux (with singularity) or OSX (with docker) with datalad-container installed via

git clone https://github.com/ReproNim/containers/
cd containers
bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')

What version of fMRIPrep are you running?

none

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

No response

Additional information / screenshots

No response

[effigies]

Duplicate of #1004.

[yarikoptic]

oh cool, I did search for Exception raised while executing Node synthstrip so failed to find that one . hopefully it gets tracked down soon. For now I guess I would need to switch to use some other app unless you are aware of some workaround? (E.g. generate some large swap file and add it to the instance)

[effigies]

Oscar can clarify, but my understanding is that the plan is to keep on with synthstrip and users will just need to throw bigger machines at the problem until some combination of numpy/pytorch/FreeSurfer take steps to reduce the memory usage.

I don't know of any workarounds, and swap files seem like they'll end up performing worse than just using a pre-SynthStrip version of MRIQC.

[yarikoptic]

ok, I will use 0.16.0 then in that test indeed