search

PennLINC / fw-heudiconv

Heuristic-based Data Curation on Flywheel
BSD 3-Clause "New" or "Revised" License
6 stars 11 forks source link

Problem running fw-heudiconv - Client version does not match server version #99

Closed michaelpeer1 closed 2 years ago

michaelpeer1 commented 2 years ago

Dear Tinashe,

Thank you for looking at this. We are running a new project and trying to run the latest version of heudiconv on our data through flywheel, but we get the following error message:

Gear Name: fw-heudiconv, Gear Version: 0.2.15_0.4.3 Executor: computeepsteinlab-6a9b1155, CPU: 8 cores, Memory: 55GB, Disk: 208GB, Swap: 32GB Gear starting... INFO:fw-heudiconv-gear:==============: fw-heudiconv gear manager starting up :===============

/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6274: UserWarning: Client version 14.6.6 does not match server version 16.3.5. Please update your client version! warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version)) WARNING:Flywheel:Use "pip install flywheel-sdk~=16.3.5" to install a compatible version for this server WARNING:Flywheel:Use "pip install flywheel-sdk~=16.3.5" to install a compatible version for this server Traceback (most recent call last): File "./fw_heudiconv_run.py", line 36, in parent_container = analysis_container.parent File "/usr/local/lib/python3.7/site-packages/flywheel/models/mixins.py", line 125, in getattr return getattr(object, name) AttributeError: type object 'object' has no attribute 'parent' Gear completed unsuccessfully after running for 1.168s. Uploading results... No output files found.

I'm attaching below the code of the heuristic file, in case that helps. Thanks a lot, and best regards

Michael Peer Postdoc, Epstein Lab mpeer@sas.upenn.edu

Heuristic file code:

!/usr/bin/env python

import os

def create_key(template, outtype=('nii.gz',), annotation_classes=None): if template is None or not template: raise ValueError('Template must be a valid format string') return template, outtype, annotation_classes

Create keys

t1w = createkey( '{subject}/{session}/anat/{subject}{session}_T1w') func_localizer_run1 = createkey( '{subject}/{session}/func/{subject}{session}_task-func_localizer_run-01_bold') func_localizer_run2 = createkey( '{subject}/{session}/func/{subject}{session}_task-func_localizer_run-02_bold') object_viewing_run1_d1 = createkey( '{subject}/{session}/func/{subject}{session}_task-object_viewing_run-01_day-01_bold') object_viewing_run2_d1 = createkey( '{subject}/{session}/func/{subject}{session}_task-object_viewing_run-02_day-01_bold') object_viewing_run1_d2 = createkey( '{subject}/{session}/func/{subject}{session}_task-object_viewing_run-01_day-02_bold') object_viewing_run2_d2 = createkey( '{subject}/{session}/func/{subject}{session}_task-object_viewing_run-02_day-02_bold') jrd_run1 = createkey( '{subject}/{session}/func/{subject}{session}_task-jrd_run-01_bold') jrd_run2 = createkey( '{subject}/{session}/func/{subject}{session}_task-jrd_run-02_bold') resting_state = createkey( '{subject}/{session}/func/{subject}{session}_task-resting_state_bold') b0_phase = createkey( '{subject}/{session}/fmap/{subject}{session}_phasediff') b0_mag = createkey( '{subject}/{session}/fmap/{subject}{session}_magnitude{item}')

def infotodict(seqinfo):

last_run = len(seqinfo)

info = {t1w:[],
    func_localizer_run1:[],
    func_localizer_run2:[],
    object_viewing_run1_d1:[],
    object_viewing_run2_d1:[],
    object_viewing_run1_d2:[],
    object_viewing_run2_d2:[],
    jrd_run1:[],
    jrd_run2:[],
    jrd_run3:[],
    resting_state:[],
    navigation:[],
    b0_phase:[], b0_mag:[]}

for s in seqinfo:
    protocol = s.protocol_name.lower()

    if "mprage" in protocol:
        info[t1w].append(s.series_id)
    elif "localizer_1" in protocol:
        info[func_localizer_run1].append(s.series_id)
    elif "localizer_2" in protocol:
        info[func_localizer_run2].append(s.series_id)
    elif "objectviewing_1_d1" in protocol:
        info[object_viewing_run1_d1].append(s.series_id)
    elif "objectviewing_2_d1" in protocol:
        info[object_viewing_run2_d1].append(s.series_id)
    elif "objectviewing_1_d2" in protocol:
        info[object_viewing_run1_d2].append(s.series_id)
    elif "objectviewing_2_d2" in protocol:
        info[object_viewing_run2_d2].append(s.series_id)
    elif "jrd_1" in protocol:
        info[jrd_run1].append(s.series_id)
    elif "jrd_2" in protocol:
        info[jrd_run2].append(s.series_id)
    elif "resting" in protocol:
        info[resting_state].append(s.series_id)
    elif "b0map" in protocol and "P" in s.image_type:
        info[b0_phase].append(s.series_id)
    elif "b0map" in protocol and "M" in s.image_type:
        info[b0_mag].append(s.series_id)

return info

MetadataExtras = { b0_phase: { "EchoTime1": 0.004, "EchoTime2": 0.006 } }

IntendedFor = { b0phase: [ '{session}/func/sub-{subject}{session}_task-func_localizer_run-01bold.nii.gz', '{session}/func/sub-{subject}{session}_task-func_localizer_run-02bold.nii.gz', '{session}/func/sub-{subject}{session}_task-object_viewing_run-01_day-01bold.nii.gz', '{session}/func/sub-{subject}{session}_task-object_viewing_run-02_day-01bold.nii.gz', '{session}/func/sub-{subject}{session}_task-object_viewing_run-01_day-02bold.nii.gz', '{session}/func/sub-{subject}{session}_task-object_viewing_run-02_day-02bold.nii.gz', '{session}/func/sub-{subject}{session}_task-jrd_run-01bold.nii.gz', '{session}/func/sub-{subject}{session}_task-jrd_run-02bold.nii.gz', '{session}/func/sub-{subject}{session}_task-resting_state_bold.nii.gz', }

def ReplaceSubject(str_input): import re out = re.sub("sub-", "", str_input) return(out)

TinasheMTapera commented 2 years ago

Hey @michaelpeer1 could you try rerunning it as is, it looks like it booted up correctly but it's possible that this is just an internet connection issue at runtime. I just ran this on my end in the GUI successfully:

Gear Name: fw-heudiconv, Gear Version: 0.2.15_0.4.3
Executor: computebbl-9229c47c, CPU: 8 cores, Memory: 55GB, Disk: 208GB, Swap: 32GB
Gear starting...
INFO:fw-heudiconv-gear:==============: fw-heudiconv gear manager starting up :===============

/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6274: UserWarning: Client version 14.6.6 does not match server version 16.3.5. Please update your client version!
  warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version))
WARNING:Flywheel:Use "pip install flywheel-sdk~=16.3.5" to install a compatible version for this server
WARNING:Flywheel:Use "pip install flywheel-sdk~=16.3.5" to install a compatible version for this server
INFO:fw-heudiconv-gear:Calling fw-heudiconv with the following settings:
INFO:fw-heudiconv-gear:Project: gear_testing
INFO:fw-heudiconv-gear:Subject(s): ['86486']
INFO:fw-heudiconv-gear:Session(s): ['11196']
INFO:fw-heudiconv-gear:Heuristic: /flywheel/v0/input/heuristic/gear_testing_heuristic.py
INFO:fw-heudiconv-gear:Action: Curate
INFO:fw-heudiconv-gear:Dry run: True
INFO:fw-heudiconv-gear:Call: fw-heudiconv-curate --verbose --project gear_testing --dry-run --subject 86486 --session 11196 --heuristic /flywheel/v0/input/heuristic/gear_testing_heuristic.py
INFO:fw-heudiconv-curator:=================: fw-heudiconv curator starting up :=================

INFO:fw-heudiconv-curator:Loading heuristic file...
INFO:fw-heudiconv-curator:Heuristic loaded successfully!
INFO:fw-heudiconv-curator:Querying Flywheel server...
/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6274: UserWarning: Client version 14.6.6 does not match server version 16.3.5. Please update your client version!
  warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version))
WARNING:Flywheel:Use "pip install flywheel-sdk~=16.3.5" to install a compatible version for this server
DEBUG:fw-heudiconv-curator:Found project: gear_testing (5c8937fddf93e3002e025e2b)
DEBUG:fw-heudiconv-curator:Found sessions:
    11196 (5d96790fa54d350044b830a5)
INFO:fw-heudiconv-curator:Applying heuristic to 11196 (1/1)...
DEBUG:fw-heudiconv-curator:Found SeqInfos:
TOF_3D_multi-slab_R2: 
        [TR=0.022 TE=0.00366 shape=(142, 384, 1, -1) image_type=('ORIGINAL', 'PRIMARY', 'MIP', 'SAG', 'ND', 'NORM')] (5d967910a54d350040b4f43d)

    TOF_3D_multi-slab_R2: 
        [TR=0.022 TE=0.00366 shape=(142, 288, 1, -1) image_type=('ORIGINAL', 'PRIMARY', 'MIP', 'COR', 'ND', 'NORM')] (5d967910a54d35003eb3de9c)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(140, 140, 1104, -1) image_type=('DERIVED', 'PRIMARY', 'DIFFUSION', 'TRACEW', 'MB', 'ND', 'NORM')] (5d967911a54d350042b67d8a)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(140, 140, 1104, -1) image_type=('DERIVED', 'PRIMARY', 'DIFFUSION', 'TRACEW', 'MB', 'ND', 'NORM')] (5d967911a54d350042b67d8a)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(140, 140, 1104, -1) image_type=('DERIVED', 'PRIMARY', 'DIFFUSION', 'TRACEW', 'MB', 'ND', 'NORM')] (5d967911a54d350042b67d8a)

    : 
        [TR=-1.0 TE=-1.0 shape=(80, 80, 120, -1) image_type=()] (5d967911a54d350033aff3ea)

    : 
        [TR=-1.0 TE=-1.0 shape=(80, 80, 120, -1) image_type=()] (5d967911a54d350033aff3ea)

    TOF_3D_multi-slab_R2: 
        [TR=0.022 TE=0.00366 shape=(384, 288, 1, -1) image_type=('ORIGINAL', 'PRIMARY', 'MIP', 'TRA', 'ND', 'NORM')] (5d967910a54d35003eb3de9b)

    Localizer: 
        [TR=0.0086 TE=0.004 shape=(512, 512, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967911a54d35003eb3de9d)

    Localizer: 
        [TR=0.0086 TE=0.004 shape=(512, 512, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967911a54d35003eb3de9d)

    Localizer: 
        [TR=0.0086 TE=0.004 shape=(512, 512, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967911a54d35003eb3de9d)

    MPRAGE_TI1100_ipat2: 
        [TR=1.81 TE=0.00345 shape=(256, 192, 160, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967910a54d350040b4f43e)

    bbl1_restbold1_124mb: 
        [TR=3 TE=0.032 shape=(448, 448, 124, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM', 'MOSAIC')] (5d967911a54d35003cb2cb9a)

    ASL_3DSPIRAL_V20_GE: 
        [TR=4 TE=0.01003 shape=(64, 64, 64, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d967911a54d350039b15361)

    TOF_3D_multi-slab_R2: 
        [TR=0.022 TE=0.00366 shape=(384, 288, 74, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967910a54d35003cb2cb99)

    T2_sagittal_SPACE: 
        [TR=3.2 TE=0.408 shape=(256, 256, 176, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d967910a54d350042b67d88)

    ASL_3DSPIRAL_V20_GE: 
        [TR=4 TE=0.01003 shape=(64, 64, 2560, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d967911a54d35003cb2cb9b)

    DTI_MultiShell_topup_ref: 
        [TR=3.027 TE=0.0828 shape=(140, 140, 92, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM')] (5d967911a54d35003eb3de9e)

    : 
        [TR=-1.0 TE=-1.0 shape=(80, 80, 60, -1) image_type=()] (5d967911a54d35003eb3de9f)

    ASL_3DSPIRAL_V20_GE: 
        [TR=4 TE=0.01003 shape=(64, 64, 32, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d967911a54d350040b4f441)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d967910a54d350042b67d89)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(1400, 1400, 118, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d967911a54d350040b4f43f)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(1400, 1400, 118, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d967911a54d350040b4f43f)

    DTI_MultiShell_117dir: 
        [TR=3.027 TE=0.0828 shape=(1400, 1400, 118, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d967911a54d350040b4f43f)

    : 
        [TR=-1.0 TE=-1.0 shape=(448, 448, 231, -1) image_type=()] (5d967911a54d350040b4f440)

INFO:fw-heudiconv-curator:Processing IntendedFor fields based on heuristic file
DEBUG:fw-heudiconv-curator:Intention map: [('sub-{subject}/{session}/fmap/sub-{subject}_{session}_phasediff',
  ['{session}/dwi/sub-{subject}_{session}_acq-multiband_dwi.nii.gz',
   '{session}/func/sub-{subject}_{session}_task-rest_acq-multiband_bold.nii.gz',
   '{session}/func/sub-{subject}_{session}_task-rest_acq-singleband_bold.nii.gz',
   '{session}/func/sub-{subject}_{session}_task-fracback_acq-singleband_bold.nii.gz',
   '{session}/func/sub-{subject}_{session}_task-face_acq-singleband_bold.nii.gz'])]
INFO:fw-heudiconv-curator:Processing Medatata fields based on heuristic file
DEBUG:fw-heudiconv-curator:Metadata extras: defaultdict(<class 'list'>, {('sub-{subject}/{session}/fmap/sub-{subject}_{session}_phasediff', ('nii.gz',), None): {'EchoTime1': 0.00412, 'EchoTime2': 0.00658}})
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300000177.nii.gz
    sub-86486_ses-11196_T1w.nii.gz -> sub-86486/ses-11196/anat/sub-86486_ses-11196_T1w.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300000354.nii.gz
    sub-86486_ses-11196_T2w.nii.gz -> sub-86486/ses-11196/anat/sub-86486_ses-11196_T2w.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300004721.bval
    sub-86486_ses-11196_acq-multiband_dwi.bval -> sub-86486/ses-11196/dwi/sub-86486_ses-11196_acq-multiband_dwi.bval
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300004721.bvec
    sub-86486_ses-11196_acq-multiband_dwi.bvec -> sub-86486/ses-11196/dwi/sub-86486_ses-11196_acq-multiband_dwi.bvec
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300004721.nii.gz
    sub-86486_ses-11196_acq-multiband_dwi.nii.gz -> sub-86486/ses-11196/dwi/sub-86486_ses-11196_acq-multiband_dwi.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300005919.nii.gz
    sub-86486_ses-11196_acq-multiband_dir-j_epi.nii.gz -> sub-86486/ses-11196/fmap/sub-86486_ses-11196_acq-multiband_dir-j_epi.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300001818.nii.gz
    sub-86486_ses-11196_task-rest_acq-multiband_bold.nii.gz -> sub-86486/ses-11196/func/sub-86486_ses-11196_task-rest_acq-multiband_bold.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300001943.nii.gz
    sub-86486_ses-11196_task-rest_acq-singleband_bold.nii.gz -> sub-86486/ses-11196/func/sub-86486_ses-11196_task-rest_acq-singleband_bold.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300004569.nii.gz
    sub-86486_ses-11196_asl.nii.gz -> sub-86486/ses-11196/func/sub-86486_ses-11196_asl.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300002008.nii.gz
    sub-86486_ses-11196_m0.nii.gz -> sub-86486/ses-11196/func/sub-86486_ses-11196_m0.nii.gz
DEBUG:fw-heudiconv-curator:
1.3.12.2.1107.5.2.43.66044.30000019032116274975300004602.nii.gz
    sub-86486_ses-11196_mean-perfusion.nii.gz -> sub-86486/ses-11196/func/sub-86486_ses-11196_mean-perfusion.nii.gz
DEBUG:fw-heudiconv-curator:Ensuring all intentions apply for acquisition DTI_MultiShell_topup_ref: 1.3.12.2.1107.5.2.43.66044.30000019032116274975300005919.nii.gz
INFO:fw-heudiconv-curator:Done!
INFO:fw-heudiconv-curator:===================: Exiting fw-heudiconv curator :===================
Series not recognized!:  TOF_3D_multi-slab_R2 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000431.nii.gz
Series not recognized!:  TOF_3D_multi-slab_R2 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000433.nii.gz
Series not recognized!:  DTI_MultiShell_117dir 1.3.12.2.1107.5.2.43.66044.30000019032116274975300005826.bval
Series not recognized!:  DTI_MultiShell_117dir 1.3.12.2.1107.5.2.43.66044.30000019032116274975300005826.bvec
Series not recognized!:  DTI_MultiShell_117dir 1.3.12.2.1107.5.2.43.66044.30000019032116274975300005826.nii.gz
Series not recognized!:   1.3.12.2.1107.5.2.43.66044.30000019032116274975300000556_e1.nii.gz
Series not recognized!:   1.3.12.2.1107.5.2.43.66044.30000019032116274975300000556_e2.nii.gz
Series not recognized!:  TOF_3D_multi-slab_R2 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000435.nii.gz
Series not recognized!:  Localizer 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000016_i00001.nii.gz
Series not recognized!:  Localizer 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000016_i00002.nii.gz
Series not recognized!:  Localizer 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000016_i00003.nii.gz
Series not recognized!:  TOF_3D_multi-slab_R2 1.3.12.2.1107.5.2.43.66044.30000019032116274975300000429.nii.gz
Series not recognized!:   1.3.12.2.1107.5.2.43.66044.30000019032116274975300000617_e2_ph.nii.gz
Series not recognized!:   1.3.12.2.1107.5.2.43.66044.30000019032116274975300006151.nii.gz

INFO:fw-heudiconv-gear:Done!
INFO:fw-heudiconv-gear:================: Exiting fw-heudiconv gear manager :=================

Gear completed successfully after running for 11.553s.
Uploading results...
No output files found.
Job complete.

Alternatively, it's easier to debug when you run it locally, so maybe you could give that a try as well --> https://fw-heudiconv.readthedocs.io/en/latest/installation.html

TinasheMTapera commented 2 years ago

Hi @michaelpeer1 just checking in, any movement on this?

michaelpeer1 commented 2 years ago

Hi Tinashe - sorry, just now saw that you responded. I tried it again now and I get the exact same error message. Maybe I'm running something wrong? I'm selecting the session (all files, after previously running dcm2niix) and then "batch run gear". I will now try running it locally according to the instructions link.

michaelpeer1 commented 2 years ago

Hi - so I ran into trouble running fw-heudiconv locally: I tried following the instructions, but got stuck at downloading the flywheel CLI (I downloaded it from the Flywheel profile page but my Avast antivirus identified the fw.exe file as infected with malware, and wouldn't let me run it). What do you suggest? Thank you!

TinasheMTapera commented 2 years ago

Hey @michaelpeer1, I think I understand the problem — when running fw-heudiconv it's best to run it at the session level or at the project level, not by batching sessions using the tick boxes. i.e. do not select any session files when running, just select the session in the left hand scroller, and go directly to Run Gear. The only input you'll be able to click will be the heuristic attachment, which should be attached to your project: readthedocs

To run fw-heudiconv on the whole project (the equivalent of a batch gear on everyone), you have to instead run an analysis gear from the project level. To do this enter your project and in the top bar navigate to Analyses, and hit Run Analysis Gear. This will allow you to launch fw-heudiconv for everyone.

Unfortunately because of Flywheel's architecture, it's not possible for this particular gear to be run using their Batch jobs functionality; we built a workaround using the project level analysis feature

michaelpeer1 commented 2 years ago

Dear Tinashe - thank you! This seems to have solved the issue, and fw-heudiconv runs normally now. However, I am encountering another issue - fMRIprep is not running properly. I am attaching the log file. I ran it with my freesurfer license file, and with "use_all_sessions" marked.

Thank you for you help!

Michael

627170dc958027b29e166674_logs.txt

TinasheMTapera commented 2 years ago

Hey @michaelpeer1 tbh we don't really dedicate any resources to maintaining the gear fmriprep-fwheudiconv, I'd recommend running the standard one that comes from Flywheel (fmriprep)... they'll also be able to provide you with better support if something were to go wrong.

I'm going to close this issue since we've addressed the original question at this point!