fwheudiconv ssl errors

[ins0mniac2]

Describe the issue We are getting an SSL error when running from MacOS 11.3.1 , and a different error when running the gear. See below.

Were you running fw-heudiconv locally or from Flywheel's GUI?

• [x ] Locally
• [ x] GUI

If you ran it locally, please describe your setup:

• OS: MacOS 11.3.1 local machine

• Version [use pip show fw-heudiconv] Name: fw-heudiconv Version: 0.4.1 Summary: Use heudiconv-like heuristics for BIDS curation on flywheel Home-page: https://github.com/PennBBL/fw_heudiconv Author: Tinashe M. Tapera, Matt Cieslak, Harsha Kethineni Author-email: tinashemtapera@gmail.com License: 3-clause BSD Location: /Users/srdas/miniconda3/lib/python3.6/site-packages Requires: bids-validator, flywheel-sdk, heudiconv, pandas, pathvalidate, pytest, pytest-cov, validators Required-by:

If you ran it through the GUI, please paste the Job ID below:

• JobID: 622002ffcb72d401fc1d4fda

Please paste your heuristic below:

import os

def create_key(template, outtype=('nii.gz',), annotation_classes=None):
    if template is None or not template:
        raise ValueError('Template must be a valid format string')
    return template, outtype, annotation_classes

# structurals
t1w = create_key(
    'sub-{subject}/{session}/anat/sub-{subject}_{session}_T1w')
t1w_nodico = create_key(
    'sub-{subject}/{session}/anat/sub-{subject}_{session}_acq_nodico_T1w')
t2w = create_key(
     'sub-{subject}/{session}/anat/sub-{subject}_{session}_T2w')
t2w_nodico = create_key(
     'sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-nodico_T2w')
flair = create_key(
    'sub-{subject}/{session}/anat/sub-{subject}_{session}_FLAIR')
flair_nodico = create_key(
    'sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-nodico_FLAIR')
# task fMRI
object_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-object_run-01_bold')
object_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-object_run-02_bold')
rhyme_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-rhyme_run-01_bold')
rhyme_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-rhyme_run-02_bold')
scenemem_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-scenemem_run-01_bold')
scenemem_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-scenemem_run-02_bold')
sentence_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-sentence_run-01_bold')
sentence_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-sentence_run-02_bold')
wordgen_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-wordgen_run-01_bold')
wordgen_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-wordgen_run-02_bold')
binder_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-binder_run-01_bold')
binder_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-binder_run-02_bold')
verbgen_run1 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-verbgen_run-01_bold')
verbgen_run2 = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-verbgen_run-02_bold')
rest = create_key(
    'sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_bold')

# ASL scans
asl = create_key(
     'sub-{subject}/{session}/asl/sub-{subject}_{session}_asl')
m0 = create_key(
    'sub-{subject}/{session}/asl/sub-{subject}_{session}_MZeroScan')
mean_perf = create_key(
    'sub-{subject}/{session}/asl/sub-{subject}_{session}_CBF')

# Field maps
b0_mag = create_key(
   'sub-{subject}/{session}/fmap/sub-{subject}_{session}_magnitude{item}')
b0_phase = create_key(
   'sub-{subject}/{session}/fmap/sub-{subject}_{session}_phasediff')

def infotodict(seqinfo):

    last_run = len(seqinfo)

    info = {t1w:[], t1w_nodico:[], t2w:[], t2w_nodico:[], flair:[], flair_nodico:[], object_run1: [], object_run2: [], rhyme_run1: [],
            rhyme_run2: [], scenemem_run1: [], scenemem_run2: [], sentence_run1: [],
            sentence_run2: [], wordgen_run1: [], wordgen_run2: [], binder_run1: [],
            binder_run2:[], verbgen_run1: [], verbgen_run2: [], rest: [], asl: [],
            m0: [], mean_perf: [], b0_phase: [], b0_mag: []}

# sometimes patients struggle with a task the first time around (or something
# else goes wrong and often some tasks are repeated. This function accomodates
# the variable number of task runs
    def get_both_series(key1, key2, s):
         if len(info[key1]) == 0:
             info[key1].append(s.series_id)
         else:
             info[key2].append(s.series_id)

# this doesn't need to be a function but using it anyway for aesthetic symmetry
# with above function
    def get_series(key, s):
            info[key].append(s.series_id)

    for s in seqinfo:
        protocol = s.protocol_name.lower()
        if any(id in protocol for id in ["t1w", "t1", "mprage_t"]):
            if s.series_description.endswith("_ND"):
                get_series(t1w_nodico,s)
            else:
                get_series(t1w,s)
        elif "flair" in protocol:
            if s.series_description.endswith("_ND"):
                get_series(flair_nodico,s)
            else:
                get_series(flair,s)
        elif any(id in protocol for id in ["t2w", "t2"]):
            if s.series_description.endswith("_ND"):
                get_series(t2w_nodico,s)
            else:
                get_series(t2w,s)
        elif "object" in protocol:
            get_both_series(object_run1,object_run2,s)
        elif "rhyming" in protocol:
            get_both_series(rhyme_run1,rhyme_run2,s)
        elif "scenemem" in protocol:
            get_both_series(scenemem_run1,scenemem_run2,s)
        elif "sentence" in protocol:
            get_both_series(sentence_run1, sentence_run2, s)
        elif "wordgen" in protocol:
            get_both_series(wordgen_run1,wordgen_run2,s)
        elif "binder" in protocol:
            get_both_series(binder_run1, binder_run2,s)
        elif "verbgen" in protocol:
            get_both_series(verbgen_run1, verbgen_run2,s)
        elif "rest" in protocol:
            if s.series_description.endswith("_AP"):
                get_series(rest,s)
        elif "spiral" in protocol:
            if s.series_description.endswith("_ASL"):
                get_series(asl,s)
            elif s.series_description.endswith("_M0"):
                get_series(m0,s)
        elif "b0map" in protocol:
                if "P" in s.image_type:
                    get_series(b0_phase,s)
                elif "M" in s.image_type:
                    get_series(b0_mag,s)

    return info

MetadataExtras = {
   b0_phase: {
       "EchoTime1": 0.00412,
       "EchoTime2": 0.00658
   }
}

IntendedFor = {
    b0_phase: [
    '{session}/func/sub-{subject}_{session}_task-rest_bold.nii.gz'],

    m0: [ 'sub-{subject}/{session}/asl/sub-{subject}_{session}_asl.nii.gz' ],
}

Please paste any relevant output below -- these are command line SSL error output

sadhana:MEMORIES2 srdas$ fw-heudiconv-curate --heuristic memoriesheudiconv.py --project MEMORIES-2 --subject 11x04069 --session 11x04069x20210611xMRI --verbose
INFO: =================: fw-heudiconv curator starting up :=================
INFO: Loading heuristic file...
INFO: Heuristic loaded successfully!
INFO: Querying Flywheel server...
2022-02-28 20:27:02,755 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ’SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)“,),)’: /api/version
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ’SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)“,),)’: /api/version

<snip>

Traceback (most recent call last):
  File “/Users/srdas/miniconda3/bin/fw-heudiconv-curate”, line 8, in <module>
    sys.exit(main())
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/fw_heudiconv/cli/curate.py”, line 331, in main
    dry_run=args.dry_run)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/fw_heudiconv/cli/curate.py”, line 101, in convert_to_bids
    project_obj = client.projects.find_first(‘label=“{}“’.format(project_label))
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/finder.py”, line 65, in find_first
    return self.__find(args, kwargs, find_first=True)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/finder.py”, line 117, in __find
    results = self._func(*self._args, **kwargs)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/flywheel.py”, line 3974, in get_all_projects
    return self.projects_api.get_all_projects(**kwargs)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/api/projects_api.py”, line 2914, in get_all_projects
    (data) = self.get_all_projects_with_http_info(**kwargs)  # noqa: E501
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/api/projects_api.py”, line 3007, in get_all_projects_with_http_info
    collection_formats=collection_formats)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/api_client.py”, line 361, in call_api
    _preload_content, _request_timeout, _request_out)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/api_client.py”, line 191, in __call_api
    _request_timeout=_request_timeout)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/api_client.py”, line 382, in request
    headers=headers)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/rest.py”, line 210, in GET
    query_params=query_params)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/flywheel/rest.py”, line 189, in request
    headers=headers)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/requests/sessions.py”, line 508, in request
    resp = self.send(prep, **send_kwargs)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/requests/sessions.py”, line 618, in send
    r = adapter.send(request, **kwargs)
  File “/Users/srdas/miniconda3/lib/python3.6/site-packages/requests/adapters.py”, line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host=‘[upenn.flywheel.io](http://upenn.flywheel.io/)’, port=443): Max retries exceeded with url: /api/projects?filter=label%3D%22MEMORIES-2%22&limit=1 (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)“,),))

Add any additional context or information below Attached is the log for the gear. 622002ffcb72d401fc1d4fda_logs.txt

[TinasheMTapera]

Hey @ins0mniac2 so I've looked into this today and specifically wanna address the SSL issue when running from the command line.

What I did was make a slight update to the docker image and reran all my tests which passed here. This tells me that it's not an SSL issue within the fw-heudiconv software -- if it were, the tests would have echoed some SSL certificate being out of date like @cookpa mentioned, but that's not the case. This leads me to believe you have SSL certificates that are out of date on your personal machine, OR an issue on Flywheel's end.

A couple of things I'd like for you to try and report back:

1. Could you re-install the CLI and login with a new API key? https://docs.flywheel.io/hc/en-us/articles/360008162214
2. Could you please report your current requests python package? pip show requests
3. Could you please update your version of requests? pip install -U requests

Let me know how each of those go, and try fw-heudiconv at your command line again

[ins0mniac2]

Updating requests fixed it! Thanks so much.

Hey @ins0mniac2 so I've looked into this today and specifically wanna address the SSL issue when running from the command line.

What I did was make a slight update to the docker image and reran all my tests which passed here. This tells me that it's not an SSL issue within the fw-heudiconv software -- if it were, the tests would have echoed some SSL certificate being out of date like @cookpa mentioned, but that's not the case. This leads me to believe you have SSL certificates that are out of date on your personal machine, OR an issue on Flywheel's end.

A couple of things I'd like for you to try and report back:

1. Could you re-install the CLI and login with a new API key? https://docs.flywheel.io/hc/en-us/articles/360008162214
2. Could you please report your current requests python package? pip show requests

Name: requests Version: 2.18.4 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: /Users/srdas/miniconda3/lib/python3.6/site-packages Requires: certifi, chardet, idna, urllib3 Required-by: etelemetry, flywheel-sdk, google-api-core, panda, requests-toolbelt

3. Could you please update your version of requests? pip install -U requests

Name: requests Version: 2.27.1 Summary: Python HTTP for Humans. Home-page: https://requests.readthedocs.io Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: /Users/srdas/miniconda3/lib/python3.6/site-packages Requires: certifi, charset-normalizer, idna, urllib3 Required-by: etelemetry, flywheel-sdk, google-api-core, panda, requests-toolbelt

Let me know how each of those go, and try fw-heudiconv at your command line again

[TinasheMTapera]

Awesome, thanks!