search

PennLINC / fw-heudiconv

Heuristic-based Data Curation on Flywheel
BSD 3-Clause "New" or "Revised" License
6 stars 11 forks source link

PET Heuristic - PET Metainformation not being stored in tabulate #74

Closed isabelannwingert closed 3 years ago

isabelannwingert commented 4 years ago

Describe the issue Hello, I had a PET image that I am basing a heuristic off of to curate PET data in FlyWheel. It looks like from the log that the curation wasn't working because PET metainformation wasn't stored in tabulate. I've attached the log below and emailed the tabulate output from the entire project to Matt and Tinashe and Azeez. The subject tested was a PET subject, and I followed a BIDS proposal for curating PET data. (https://docs.google.com/document/d/1mqMLnxVdLwZjDd4ZiWFqjEAmOmfcModA_R535v3eQs0/edit#)

Were you running fw-heudiconv locally or from Flywheel's GUI?

If you ran it through the GUI, please paste the Job ID below:

Please paste your heuristic below:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Mon May 11 10:01:16 2020

@author: isabelannwingert
"""

import datetime
def create_key(template, outtype=('nii.gz',), annotation_classes=None):
    if template is None or not template:
        raise ValueError('Template must be a valid format string')
    return template, outtype, annotation_classes

# pet images
ac_dyn_av1451 = create_key('sub-{subject}/{session}/pet/sub-{subject}_{session}_task-resting_acq-AV1451_rec-acdyn_pet')
nac_dyn_av1451 = create_key('sub-{subject}/{session}/pet/sub-{subject}_{session}_task-resting_acq-AV1451_rec-nacdyn_pet')
ac_stat_av1451 = create_key('sub-{subject}/{session}/pet/sub-{subject}_{session}_task-resting_acq-AV1451_rec-acstat_pet')
nac_stat_av1451 = create_key('sub-{subject}/{session}/pet/sub-{subject}_{session}_task-resting_acq-AV45_rec-nacstat_pet')

from collections import defaultdict
def infotodict(seqinfo):
    """Heuristic evaluator for determining which runs belong where
        allowed template fields - follow python string module:
        item: index within category
        subject: participant id
        seqitem: run number during scanning
        subindex: sub index within group"""

    info = { ac_dyn_av1451: [], 
            nac_dyn_av1451: [], 
            ac_stat_av1451: [], 
            nac_stat_av1451: [] }

    for s in seqinfo:
        protocol = s.protocol_name.lower()
        if "DY_CTAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[ac_dyn_av1451].append(s.series_id)
        elif "DY_NAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[nac_dyn_av1451].append(s.series_id)
        elif "[BR_CTAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[ac_stat_av1451].append(s.series_id)
        elif "[BR_NAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[nac_stat_av1451].append(s.series_id)
        else:
            print("Series not recognized!: ", protocol, s.dcm_dir_name)

    return info, seqinfo

Please paste any relevant output below LOG IS DOWN BELOW:

Gear Name: fw-heudiconv, Gear Version: 0.2.11-1_0.2.5
Gear starting...

/flywheel/v0/fw_heudiconv/backend_funcs/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: ==============: fw-heudiconv gear manager starting up :===============

/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6033: UserWarning: Client version 11.4.5 does not match server version 12.0.1. Please update your client version!
  warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version))
WARNING: Use "pip install flywheel-sdk~=12.0.1" to install a compatible version for this server
INFO: Calling fw-heudiconv with the following settings:
INFO: Project: NACC-SC
INFO: Subject(s): ['125944']
INFO: Session(s): ['125944_AV1451PET_20200310']
INFO: Heuristic: /flywheel/v0/input/heuristic/PET_heuristic_draft.py
INFO: Action: Curate
INFO: Dry run: True
/usr/local/lib/python3.7/site-packages/fw_heudiconv/backend_funcs/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: =================: fw-heudiconv curator starting up :=================

INFO: Loading heuristic file...
INFO: Heuristic loaded successfully!
INFO: Querying Flywheel server...
/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6033: UserWarning: Client version 11.4.5 does not match server version 12.0.1. Please update your client version!
  warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version))
WARNING: Use "pip install flywheel-sdk~=12.0.1" to install a compatible version for this server
DEBUG: Found project: NACC-SC (5c508d5fc2a4ad002d7628d8)
DEBUG: Found sessions:
    125944_AV1451PET_20200310 (5e85f255cdb5ca438d09cdf7)
INFO: Applying heuristic to 125944_AV1451PET_20200310 (1/1)...
DEBUG: Found SeqInfos:
Brain-nac-2mm.xml: 
        [TR=-1.0 TE=-1.0 shape=(128, 128, 90, -1) image_type=('ORIGINAL', 'PRIMARY')] (5e85f258cdb5ca438d09ce71)

    Brain-nac-2mm.xml: 
        [TR=-1.0 TE=-1.0 shape=(128, 128, 90, -1) image_type=('ORIGINAL', 'PRIMARY')] (5e85f259cdb5ca438d09cee2)

    Brain-nac-2mm.xml: 
        [TR=-1.0 TE=-1.0 shape=(128, 128, 540, -1) image_type=('ORIGINAL', 'PRIMARY')] (5e85f262cdb5ca438d09d0cb)

    Brain-nac-2mm.xml: 
        [TR=-1.0 TE=-1.0 shape=(128, 128, 540, -1) image_type=('ORIGINAL', 'PRIMARY')] (5e85f269cdb5ca438d09d211)

Series not recognized!:  brain-nac-2mm.xml BR_NAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR_CTAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR-DY_NAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR-DY_CTAC AV1451 Brain Dynamic.nii.gz

Traceback (most recent call last):
  File "/usr/local/bin/fw-heudiconv-curate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/fw_heudiconv/cli/curate.py", line 245, in main
    dry_run=args.dry_run)
  File "/usr/local/lib/python3.7/site-packages/fw_heudiconv/cli/curate.py", line 159, in convert_to_bids
    for key, val in to_rename.items():
AttributeError: 'tuple' object has no attribute 'items'
INFO: Done!
INFO: ================: Exiting fw-heudiconv gear manager :=================

Gear completed unsuccessfully after running for 8.828s.
Uploading results...

Add any additional context or information below

TinasheMTapera commented 4 years ago

Hey @isabelannwingert sorry for the delay. I think the problem is that your heuristic is unpacking two items instead of one:

def infotodict(seqinfo):
    """Heuristic evaluator for determining which runs belong where
        allowed template fields - follow python string module:
        item: index within category
        subject: participant id
        seqitem: run number during scanning
        subindex: sub index within group"""

    info = { ac_dyn_av1451: [], 
            nac_dyn_av1451: [], 
            ac_stat_av1451: [], 
            nac_stat_av1451: [] }

    for s in seqinfo:
        protocol = s.protocol_name.lower()
        if "DY_CTAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[ac_dyn_av1451].append(s.series_id)
        elif "DY_NAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[nac_dyn_av1451].append(s.series_id)
        elif "[BR_CTAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[ac_stat_av1451].append(s.series_id)
        elif "[BR_NAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[nac_stat_av1451].append(s.series_id)
        else:
            print("Series not recognized!: ", protocol, s.dcm_dir_name)

    return info, seqinfo                            #### THIS SHOULD ONLY RETURN <info>
isabelannwingert commented 4 years ago

Hi!

That didn't work, although no errors were raised.

I did get a message that the PET series I was trying to curate were not recognized.

Series not recognized!:  brain-nac-2mm.xml BR_NAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR_CTAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR-DY_NAC AV1451 Brain Dynamic.nii.gz
Series not recognized!:  brain-nac-2mm.xml BR-DY_CTAC AV1451 Brain Dynamic.nii.gz
TinasheMTapera commented 4 years ago

Ok I looked at the seqinfo table you sent via email, I think you may be approaching this incorrectly. Here's a few problems:

First, you've assigned a variable protocol that is supposed to be the protocol name in lowercase, and then you don't use it for your if statements:

    for s in seqinfo:
        protocol = s.protocol_name.lower()
        if "DY_CTAC] AV1451 Brain Dynamic" in s.protocol_name:
            info[ac_dyn_av1451].append(s.series_id)

Secondly, remember that the in reserved word in Python is explicit for the range of data you search for. For example, 'oo' in 'foobar' is true but 'oo a' in foobar is false. Your use of in in your conditionals is assuming that the string s.protocol_name contains something like "foooo*****DY_CTAC] AV1451 Brain Dynamic******bar". I just don't think this is true, because s.protocol_name cannot have spaces in it, and I doubt there are any protocols I've come across that use the bracket symbol.

Remember that s.protocol_name comes straight out of the seqinfo table's protocol_name column, in which I didn't see any strings like the one you've used.

I think it would be helpful for you to review the tutorial and take another shot.

isabelannwingert commented 3 years ago

Hi Tinashe, the issue here has been resolved, and PET data can be curated now. However, there are a few bugs that I would like to raise that is related to this. I will continue on the discussion here later today with what happened during curation. Thanks!