--cifti-output error singularity version 20.0.0

[saigerutherford]

Cifti output spaces does not seem to be working in the singularity container version 20.0.0.

Here is the command I am running fMRIPrep with:

FMRIPREP=/sw/examples/singularity/fmriprep-20.0.0.simg
SURF_LICENSE=${SCRIPTDIR}/data/fs_license.txt

BIDS_DIR=/tmp/workflow_usefs_${SUB}/BIDS
OUTPUT_DIR=/tmp/workflow_usefs_${SUB}/derivatives
WORK_DIR=/tmp/workflow_usefs_${SUB}/work

singularity run --cleanenv $FMRIPREP                        \
    $BIDS_DIR $OUTPUT_DIR participant                       \
    --nthreads ${NTHREADS}                                  \
    --mem-mb ${MEMMB}                                       \
    --fs-license-file=$SURF_LICENSE                         \
    --fs-subjects-dir ${OUTPUT_DIR}/freesurfer              \
    --participant-label=sub-${SUB}                          \
    -w $WORK_DIR                                            \
    --ignore slicetiming                                    \
    --cifti-output                                         \
    --use-aroma                                             \
    --aroma-melodic-dimensionality -100                     \
    --output-spaces anat fsLR MNI152NLin6Asym:res-2

And here is the error output:

        Summary:                Available Tasks:        Available Modalities: 
        25 Files, 3.55GB        task-gaze               T1w                   
        1 - Subject             task-jovi               T2w                   
        1 - Session             task-rest               bold                  
                                                        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
200225-12:51:44,324 nipype.workflow IMPORTANT:

    Running fMRIPREP version 20.0.0:
      * BIDS dataset path: /tmp/workflow_usefs_2060/BIDS.
      * Participant list: ['2060'].
      * Run identifier: 20200225-125144_8a8b4fba-3af8-416a-ba3b-083451d3c2ee.

    Spatial References:
    - Reference(space='anat', spec={})
    - Reference(space='fsLR', spec={})
    - Reference(space='MNI152NLin6Asym', spec={'res': '2'})

200225-12:51:49,779 nipype.utils WARNING:
     Could not check for version updates: 
Connection to server could not be made
200225-12:51:51,224 nipype.workflow IMPORTANT:
     Creating bold processing workflow for "/tmp/workflow_usefs_2060/BIDS/sub-2060/func/sub-2060_task-gaze_run-01_bold.nii" (0.28 GB / 276 TRs). Memory resampled/largemem=1.14/1.92 GB.
200225-12:51:51,232 nipype.workflow IMPORTANT:
     No single-band-reference found for sub-2060_task-gaze_run-01_bold.nii
200225-12:51:51,644 nipype.interface INFO:
     We advise you to upgrade DIPY version. This upgrade will open access to more function
200225-12:51:51,646 nipype.interface INFO:
     We advise you to upgrade DIPY version. This upgrade will open access to more function
200225-12:51:51,647 nipype.interface INFO:
     We advise you to upgrade DIPY version. This upgrade will open access to more models
200225-12:51:52,534 nipype.workflow IMPORTANT:
     Creating BOLD surface-sampling workflow.
Downloading https://templateflow.s3.amazonaws.com/tpl-fsLR/tpl-fsLR_space-fsaverage_hemi-L_den-164k_sphere.surf.gii
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x2b15b0a61518>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-fsLR/tpl-fsLR_space-fsaverage_hemi-L_den-164k_sphere.surf.gii (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2b15b0a61518>: Failed to establish a new connection: [Errno 110] Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 685, in build_workflow
    bids_filters=bids_filters,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 274, in init_fmriprep_wf
    bids_filters=bids_filters,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 638, in init_single_subject_wf
    use_syn=use_syn,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/bold/base.py", line 946, in init_func_preproc_wf
    repetition_time=metadata['RepetitionTime'])
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/bold/resampling.py", line 743, in init_bold_grayords_wf
    for hemi in 'LR'
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/bold/resampling.py", line 743, in <listcomp>
    for hemi in 'LR'
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 67, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 178, in _s3_get
    r = requests.get(url, stream=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-fsLR/tpl-fsLR_space-fsaverage_hemi-L_den-164k_sphere.surf.gii (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2b15b0a61518>: Failed to establish a new connection: [Errno 110] Connection timed out'))

[mgxd]

Looks like you are hitting a problem fetching the templates - could you take a look at https://fmriprep.readthedocs.io/en/stable/singularity.html#internet-access-problems and see if pre-fetching locally helps?

In any case, it may be worthwhile to include the fsLR templates directly in the Docker image to avoid this hassle - any objections @oesteban @effigies ?

[effigies]

@mgxd You covered everything I was about to type. Yes, let's do that.

[saigerutherford]

Thanks for addressing this so quickly! I tried to go down the road of fetching the templates on my own but got stuck when transferring it over to the cluster. How can I tell when a new Docker image with the templates included will be ready to build?

[mgxd]

It will be a new release (20.0.1)

[justbennet]

I take it, it will be a bit before the 20.0.1 release hits the street, yes? But in the meantime, I think this would make the Singularity container created from 20.0.0 workable now, yes?

Finally, run the singularity image binding the appropriate folder:

$ export SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow
$ singularity run -B ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}:/templateflow \
--cleanenv fmriprep.simg <fmriprep arguments>

and that would bind the /templateflow inside the container to the invoking user's $HOME/.cache/templateflow directory, if I'm reading that correctly...?

[oesteban]

and that would bind the /templateflow inside the container to the invoking user's $HOME/.cache/templateflow directory, if I'm reading that correctly...?

unless I'm missing some detail, yes - this is correct. Your singularity installation should honor the SINGULARITYENV_* magic though.

If not, you can just do:

$ singularity run -B ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}:/home/fmriprep/.cache/templateflow \
      --cleanenv fmriprep.simg <fmriprep arguments>

[effigies]

This was fixed in #1996, and #1997 should ensure that we catch these problems in the future, at least for common templates.

I think we can have a bug-fix release by next Monday. Possibly earlier, but it's a busy week so I don't want to make promises others might need to scramble to keep. :-)