Closed JuYuqian closed 3 years ago
effigies
commented
3 years ago I haven't seen anything like this before. It appears to be synchronization issue, but it's internal to the Python multiprocessing API, so I'm not sure how we would fix it.
Does this happen reliably?
JuYuqian
commented
3 years ago Yes, I tried multiple times. and changed the subject labels, the issue still exists.
effigies
commented
3 years ago I'm going to (reluctantly) ping @oesteban. There's something nasty going on with process managers, but Googling and a little bit of digging into CPython is not buying me much clarity.
oesteban
commented
3 years ago Perhaps overriding the use of forkserver?
mgxd
commented
3 years ago I wonder if we even need to set the multiprocessing start method anymore, now that we can read/write to the config file.
oesteban
commented
3 years ago I wonder if we even need to set the multiprocessing start method anymore, now that we can read/write to the config file.
Yes, we still need to do this.
The reason being that we want to start the forkserver as soon as possible in the runtime, when the memory fingerprint is minimal (typically ~850MB at that point on a Linux box). We want to use the forkserver because it starts as many workers as you request, and when the workers are cleansed (and replaced with a fresh one), they start with the same memory fingerprint of the original fork (remember, ~850MB for fMRIPrep when the forkserver is initiated).
On Mac this should not be necessary in principle. 1) I think mac uses spawn to start subprocesses, so there's no step where virtual memory blows up when forking; 2) even if it used fork, the problem comes from how CPython handles memory in Linux (it does not release memory from previously used imports, which is not the case in Windows and, probably -I'm not sure- in mac).
Long story short, forkserver is still necessary on Linux.
effigies
commented
3 years ago The problem is that they're using docker on Mac, so we can't dispense with forkserver. The underlying Python is still Linux.
oesteban
commented
3 years ago Ugh - thought it was baremetal installation.
JuYuqian
commented
3 years ago Sorry about confusion. I'm using the Docker Wrapper.
effigies
commented
3 years ago @JuYuqian, I think I have a fix in #2435. Would you give the following a try?
git clone https://github.com/nipreps/fmriprep.git
git -C fmriprep fetch origin pull/2435/head
git -C fmriprep checkout FETCH_HEADYou should see something like:
Cloning into 'fmriprep'...
remote: Enumerating objects: 30475, done.
remote: Counting objects: 100% (242/242), done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 30475 (delta 143), reused 206 (delta 127), pack-reused 30233
Receiving objects: 100% (30475/30475), 40.23 MiB | 14.21 MiB/s, done.
Resolving deltas: 100% (22401/22401), done.
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (1/1), done.
remote: Total 5 (delta 4), reused 5 (delta 4), pack-reused 0
Unpacking objects: 100% (5/5), 717 bytes | 717.00 KiB/s, done.
From https://github.com/nipreps/fmriprep
* branch refs/pull/2435/head -> FETCH_HEAD
Note: switching to 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 314dc39a FIX: Fall-back to initializing workflow in main processThen you can patch this into the Docker container with fmriprep-docker as follows:
fmriprep-docker --patch fmriprep=fmriprep/fmriprep [...]Where [...] is the rest of your usual command.
JuYuqian
commented
3 years ago Thank you, @effigies ,
After 'cloning into 'fmriprep......'', I tried these two commands, but neither works. Maybe I typed the command in a wrong way?
fmriprep-docker --patch fmriprep=fmriprep/fmriprep /Users/yuqianni/Desktop/affective_video_raw_data /Users/yuqianni/Desktop/affective_video_preprocessing participant --participant-label 01 --fs-license-file /Users/yuqianni/Freesurfer/freesurfer.txt
fmriprep-docker --patch fmriprep=fmriprep/fmriprep fmriprep-docker /Users/yuqianni/Desktop/affective_video_raw_data /Users/yuqianni/Desktop/affective_video_preprocessing participant --participant-label 01 --fs-license-file /Users/yuqianni/Freesurfer/freesurfer.txt
Both command gave me this error:
Traceback (most recent call last):
File "/Users/yuqianni/.local/bin/fmriprep-docker", line 8, in <module>
sys.exit(main())
File "/Users/yuqianni/.local/lib/python3.8/site-packages/fmriprep_docker.py", line 349, in main
opts, unknown_args = parser.parse_known_args()
File "/Users/yuqianni/opt/anaconda3/lib/python3.8/argparse.py", line 1800, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Users/yuqianni/opt/anaconda3/lib/python3.8/argparse.py", line 2006, in _parse_known_args
start_index = consume_optional(start_index)
File "/Users/yuqianni/opt/anaconda3/lib/python3.8/argparse.py", line 1946, in consume_optional
take_action(action, args, option_string)
File "/Users/yuqianni/opt/anaconda3/lib/python3.8/argparse.py", line 1874, in take_action
action(self, namespace, argument_values, option_string)
File "/Users/yuqianni/.local/lib/python3.8/site-packages/fmriprep_docker.py", line 236, in __call__
k, v = kv.split("=")
ValueError: not enough values to unpack (expected 2, got 1)Oh, you may need to put the patch flag at the end, not the beginning.
JuYuqian
commented
3 years ago Thank you. It started successfully. I'll let it run over night to see if it can complete sucessfully.
Here is what I see so far:
(base) yuqianni@Yuqians-Mac-mini ~ % fmriprep-docker /Users/yuqianni/Desktop/affective_video_raw_data /Users/yuqianni/Desktop/affective_video_preprocessing participant --participant-label 01 --fs-license-file /Users/yuqianni/Freesurfer/freesurfer.txt --fd-spike-threshold 0.5 --dvars-spike-threshold 1.5 --patch fmriprep=fmriprep/fmriprep
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Warning: <8GB of RAM is available within your Docker environment.
Some parts of fMRIPrep may fail to complete.
Continue anyway? [y/N]y
RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=20.10.7 -it -v /Users/yuqianni/fmriprep/fmriprep:/usr/local/miniconda/lib/python3.7/site-packages/fmriprep:ro -v /Users/yuqianni/Freesurfer/freesurfer.txt:/opt/freesurfer/license.txt:ro -v /Users/yuqianni/Desktop/affective_video_raw_data:/data:ro -v /Users/yuqianni/Desktop/affective_video_preprocessing:/out nipreps/fmriprep:20.2.1 /data /out participant --participant-label 01 --fd-spike-threshold 0.5 --dvars-spike-threshold 1.5
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
You are using fMRIPrep-0+unknown, and a newer version of fMRIPrep is available: 20.2.1.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
bids-validator@1.4.0
This dataset appears to be BIDS compatible.
Summary: Available Tasks: Available Modalities:
56 Files, 2.77GB Functional localizer T1w
11 - Subjects Passive Viewing bold
1 - Session
If you have any questions, please post on https://neurostars.org/tags/bids.
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
210702-04:34:32,1 nipype.workflow IMPORTANT:
Running fMRIPREP version 0+unknown:
* BIDS dataset path: /data.
* Participant list: ['01'].
* Run identifier: 20210702-043401_8ff1d701-e9a8-4ec1-9607-1a2fbe52c5cb.
* Output spaces: MNI152NLin2009cAsym:res-native.
* Pre-run FreeSurfer's SUBJECTS_DIR: /out/freesurfer.
210702-04:34:34,160 nipype.workflow INFO:
No single-band-reference found for sub-01_task-functionallocalizer_run-01_bold.nii.gz.
210702-04:34:35,900 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
210702-04:34:36,239 nipype.workflow INFO:
No single-band-reference found for sub-01_task-view_run-01_bold.nii.gz.
210702-04:34:37,702 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
210702-04:34:38,32 nipype.workflow INFO:
No single-band-reference found for sub-01_task-view_run-02_bold.nii.gz.
210702-04:34:39,497 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
Process Process-2:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/workflow.py", line 82, in build_workflow
retval["workflow"] = init_fmriprep_wf()
File "<string>", line 2, in __setitem__
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod
kind, result = conn.recv()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
210702-04:34:46,690 nipype.workflow WARNING:
Could not build workflow in separate process. Attempting to build in main process. This may impact memory footprint.
210702-04:34:51,883 nipype.workflow IMPORTANT:
Running fMRIPREP version 0+unknown:
* BIDS dataset path: /data.
* Participant list: ['01'].
* Run identifier: 20210702-043401_8ff1d701-e9a8-4ec1-9607-1a2fbe52c5cb.
* Output spaces: MNI152NLin2009cAsym:res-native.
* Pre-run FreeSurfer's SUBJECTS_DIR: /out/freesurfer.
210702-04:34:55,318 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
210702-04:34:56,782 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
210702-04:34:58,240 nipype.workflow IMPORTANT:
Slice-timing correction will be included.
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
Could not generate CITATION.html file:
pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle="fMRIPrep citation boilerplate" /out/fmriprep/logs/CITATION.md -o /out/fmriprep/logs/CITATION.html
Could not generate CITATION.tex file:
pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --natbib /out/fmriprep/logs/CITATION.md -o /out/fmriprep/logs/CITATION.tex
210702-04:35:20,218 nipype.workflow IMPORTANT:
fMRIPrep started!
210702-04:36:20,938 nipype.workflow WARNING:
Some nodes exceed the total amount of memory available (1.75GB).It still doesn't work, unfortunately. After letting it running over night, this is what I got:
210702-04:35:20,218 nipype.workflow IMPORTANT:
fMRIPrep started!
210702-04:36:20,938 nipype.workflow WARNING:
Some nodes exceed the total amount of memory available (1.75GB).
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
210702-04:36:33,318 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_functionallocalizer_run_01_wf.func_derivatives_wf.raw_sources" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_functionallocalizer_run_01_wf/func_derivatives_wf/raw_sources".
210702-04:36:33,322 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_01_wf.func_derivatives_wf.raw_sources" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_01_wf/func_derivatives_wf/raw_sources".
210702-04:36:33,333 nipype.workflow INFO:
[Node] Running "raw_sources" ("nipype.interfaces.utility.wrappers.Function")
210702-04:36:33,339 nipype.workflow INFO:
[Node] Running "raw_sources" ("nipype.interfaces.utility.wrappers.Function")
210702-04:36:33,356 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_functionallocalizer_run_01_wf.func_derivatives_wf.raw_sources".
210702-04:36:33,363 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_01_wf.func_derivatives_wf.raw_sources".
210702-04:36:35,920 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_02_wf.func_derivatives_wf.raw_sources" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_02_wf/func_derivatives_wf/raw_sources".
210702-04:36:35,929 nipype.workflow INFO:
[Node] Running "raw_sources" ("nipype.interfaces.utility.wrappers.Function")
210702-04:36:35,946 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_02_wf.func_derivatives_wf.raw_sources".
210702-04:36:36,836 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.brain_extraction_wf.lap_tmpl" in "/tmp/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/brain_extraction_wf/lap_tmpl".
210702-04:36:36,845 nipype.workflow INFO:
[Node] Running "lap_tmpl" ("nipype.interfaces.ants.utils.ImageMath"), a CommandLine Interface with command:
ImageMath 3 tpl-OASIS30ANTs_res-01_T1w_maths.nii.gz Laplacian /home/fmriprep/.cache/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz 1.5 1
210702-04:36:38,691 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_01_wf.initial_boldref_wf.val_bold" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_01_wf/initial_boldref_wf/val_bold".
210702-04:36:38,697 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_functionallocalizer_run_01_wf.initial_boldref_wf.val_bold" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_functionallocalizer_run_01_wf/initial_boldref_wf/val_bold".
210702-04:36:38,731 nipype.workflow INFO:
[Node] Setting-up "_val_bold0" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_functionallocalizer_run_01_wf/initial_boldref_wf/val_bold/mapflow/_val_bold0".
210702-04:36:38,731 nipype.workflow INFO:
[Node] Setting-up "_val_bold0" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_01_wf/initial_boldref_wf/val_bold/mapflow/_val_bold0".
210702-04:36:38,741 nipype.workflow INFO:
[Node] Running "_val_bold0" ("niworkflows.interfaces.images.ValidateImage")
210702-04:36:38,746 nipype.workflow INFO:
[Node] Running "_val_bold0" ("niworkflows.interfaces.images.ValidateImage")
210702-04:36:39,83 nipype.workflow INFO:
[Node] Finished "_val_bold0".
210702-04:36:39,83 nipype.workflow INFO:
[Node] Finished "_val_bold0".
210702-04:36:39,90 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_functionallocalizer_run_01_wf.initial_boldref_wf.val_bold".
210702-04:36:39,95 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_01_wf.initial_boldref_wf.val_bold".
210702-04:36:40,521 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_02_wf.initial_boldref_wf.val_bold" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_02_wf/initial_boldref_wf/val_bold".
210702-04:36:40,532 nipype.workflow INFO:
[Node] Setting-up "_val_bold0" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_view_run_02_wf/initial_boldref_wf/val_bold/mapflow/_val_bold0".
210702-04:36:40,536 nipype.workflow INFO:
[Node] Running "_val_bold0" ("niworkflows.interfaces.images.ValidateImage")
210702-04:36:40,695 nipype.workflow INFO:
[Node] Finished "_val_bold0".
210702-04:36:40,703 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_task_view_run_02_wf.initial_boldref_wf.val_bold".
210702-04:36:40,890 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.func_preproc_task_functionallocalizer_run_01_wf.initial_boldref_wf.gen_ref" in "/tmp/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_functionallocalizer_run_01_wf/initial_boldref_wf/gen_ref".
210702-04:36:40,904 nipype.workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.registration.EstimateReferenceImage")
210702-04:36:41,498 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.brain_extraction_wf.full_wm" in "/tmp/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/brain_extraction_wf/full_wm".
210702-04:36:41,516 nipype.workflow INFO:
[Node] Running "full_wm" ("nipype.interfaces.utility.wrappers.Function")
210702-04:36:43,752 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.brain_extraction_wf.full_wm".
210702-04:36:44,605 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.brain_extraction_wf.res_tmpl" in "/tmp/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/brain_extraction_wf/res_tmpl".
210702-04:36:44,832 nipype.workflow INFO:
[Node] Running "res_tmpl" ("niworkflows.interfaces.images.RegridToZooms")
210702-04:36:49,515 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_01_wf.bidssrc" in "/tmp/work/fmriprep_wf/single_subject_01_wf/bidssrc".
exception calling callback for <Future at 0x4060038898 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
210702-04:36:49,643 nipype.workflow INFO:
[Node] Running "bidssrc" ("niworkflows.interfaces.bids.BIDSDataGrabber")
210702-04:36:49,648 nipype.interface INFO:
No "t2w" images found for sub-01
210702-04:36:49,649 nipype.interface INFO:
No "flair" images found for sub-01
210702-04:36:49,649 nipype.interface INFO:
No "fmap" images found for sub-01
210702-04:36:49,649 nipype.interface INFO:
No "sbref" images found for sub-01
210702-04:36:49,650 nipype.interface INFO:
No "roi" images found for sub-01
exception calling callback for <Future at 0x406129fa58 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x40612e9e48 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
210702-04:36:49,707 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_01_wf.bidssrc".
exception calling callback for <Future at 0x405ef0dfd0 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.That error usually occurs when you don't have enough memory - can you try upping the amount of memory available to Docker? https://stackoverflow.com/a/44533437
effigies
commented
3 years ago And it looks like the bug fix works, so I'll go ahead and merge it.
JuYuqian
commented
3 years ago I increased memory in Docker and fmriprep has been running for a while without error. Thank you so much for the help. I'll update again once it finishes.
JuYuqian
commented
3 years ago Here is an update:
I allocated 6 CPUs and 12 GB memory to docker on my M1 chip Mac mini to process one subject's data (2 runs, each run has 365 volumes). I tried twice with --fs-no-reconall. The first time, it failed with a memory error. The second time, it has been running for about 20 hours, but still hasn't finished yet, no error though. It hang with the following message for more than 8 hours:
210704-06:34:17,299 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_05_wf.func_preproc_task_functionallocalizer_run_01_wf.bold_confounds_wf.spike_regressors".I ran the same preprocess with --fs-no-reconall on a Windows laptop with 4 cores and 8 GB memory, it took about 4 hours to complete successfully.
I also tried preprocess the same subject's data using the M1 Mac mini with reconall included. This time, it hang with the following for more than 8 hours:
[Node] Finished "fmriprep_wf.single_subject_03_wf.anat_preproc_wf.surface_recon_wf.t1w2fsnative_xfm"
So after letting it running for about 20 hours in total, I stopped it. The same process can usually be finished with my 8GB memory Windows laptop within 20 hours.
So seems like running the Docker image on M1 chip Mac mini is much slower than running it on Windows PC, even the Mac Mini has more CPU and memory allocated to Docker. Maybe partially because the Docker image is linux/amd64 whereas the new Mac mini is arm? I saw a discussion talking about running amd64 images on M1 Mac may run into memory allocation problems: https://github.com/docker/for-mac/issues/5204#issuecomment-826629491
effigies
commented
3 years ago Interesting. Thanks for the update, and that makes some sense. I can see a few ways forward, in order of easiest to coolest:
1) Run fMRIPrep on x86_64 chips. 2) Set up a native installation on your M1. 3) Work on building an ARM-based Docker image. If you can get this to a solid point, we could start publishing ARM tags to dockerhub.
If we do start supporting multiple architectures, we'll need to add that to the reporting.
JuYuqian
commented
3 years ago Thank you. I may try to set up a native installation on my M1 some time. But I'm not really a programmer so I lack the skill to build an ARM-based Docker image (really hope I have the skill though). Still, I agree that supporting multiple architecture is a great way moving forward.
effigies
commented
3 years ago Sounds good. I think we can close this for now. If you start work on a native installation or Docker image and need some guidance, then we can either reopen or start a new issue.
JuYuqian
commented
3 years ago Sure, thanks a lot for the help!
stebo85
commented
1 year ago Just one little addition to this thread: When activating Rosetta in Docker one can get a decent performance with x86 images on ARM:
I was trying to use fmriprep on my M1 chip Mac Mini with Docker Wrapper. After I typed in the command in terminal, it started to test BIDS compatibility. It went well till saying 'Slic-timing correction will be included', then error messages pop up.
I wonder if it's a M1 chip issue or I did something wrong. I actually successfully did the same preprocessing using fmriprep on another Windows 10 Laptop.
Any help is appreciated.
Below is my command and the error messages: