Heuristic sorts data all underneath `anat` container, despite indentations being consistent

[isabelannwingert]

GitHub doesn't support attaching .py scripts, so heuristic is down below:

def create_key(template, outtype=('nii.gz',), annotation_classes=None):
    if template is None or not template:
        raise ValueError('Template must be a valid format string')
    return template, outtype, annotation_classes

#for s in seqinfo:
    #series_desc = s.series_description.lower()

### anatomical images - t1 (identifier = t1)
t1w_axial = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-axial_T1w')
t1w_sagittal = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-sagittal_T1w')
t1w_etc = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-{series_desc}_T1w')
t2w_axial = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-axial_T2w')
t2w_sagittal = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-sagittal_T2w')
t2w_etc = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-{series_desc}_T2w')

### anatomical images - flair (identifier = flair)
flair_axial = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-axial_FLAIR')
flair_sagittal = create_key('sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-sagittal_FLAIR')

### ---

### functional images / runs / field maps (identifier = rsfmri)
rsfmri_axial = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_acq-axial_bold')
rs_ap_axial = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_acq-axial_dir-ap_run-1_bold')
rs_pa_axial = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_acq-axial_dir-pa_run-2_bold')
rs_etc = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_acq-{series}_bold')
rs_pa_etc = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_acq-{series}_dir-pa_run-2_bold')

### resting_bold (identifier = resting)
#resting_bold = create_key('sub-{subject}/{session}/func/sub-{subject}_{session}_task-rest_bold') --> look @ tabulate

### ---

### diffusion/dwi images - bo (identifier = b0)
#mag1 = create_key('sub-{subject}/{session}/fmap/sub-{subject}_{session}_magnitude{item}')
#phase_diff = create_key('sub-{subject}/{session}/fmap/sub-{subject}_{session}_phasediff')

### diffusion images (identifier = dti)
dti_axial = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-axial_dwi')
dti_ap_axial = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-axial_dir-ap_dwi')
dti_34dir = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-34dir_dwi')
dti_30dir = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-30dir_dwi')
dti_etc = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-{series}_dwi')
dti_pa_etc = create_key('sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-{series}_dir-ap_dwi')
### ---

### localizer (identifier = localizer) ---> Phil confirmed to keep it as 'localizer'
localizer = create_key('sub-{subject}/{session}/localizer/sub-{subject}_{session}_localizer')

### ASL (identifier = ep2) ---> will standardize soon
#ep2 = create_key('sub-{subject}/{session}/ASL/sub-{subject}_{session}_ASL_nonM0Co')
#ep2_moco = create_key('sub-{subject}/{session}/ASL/sub-{subject}_{session}_ASL_M0Co')

from collections import defaultdict
def infotodict(seqinfo):
    """Heuristic evaluator for determining which runs belong where
        allowed template fields - follow python string module:
        item: index within category
        subject: participant id
        seqitem: run number during scanning
        subindex: sub index within group"""

    info = {
        t1w_axial: [], t1w_sagittal: [], t1w_etc: [], t2w_axial: [], t2w_sagittal: [], t2w_etc: [], flair_axial: [],
        flair_sagittal: [],rsfmri_axial: [], rs_ap_axial: [], rs_pa_axial: [], rs_etc: [], rs_pa_etc: [], dti_axial: [],
        dti_ap_axial: [], dti_34dir: [], dti_30dir: [], dti_etc: [], dti_pa_etc: [], localizer: []
    }

    for s in seqinfo:
        protocol = s.protocol_name.lower()
        series_desc = s.series_description.lower()
    ### /anat/t1w ###
        if "t1_mpr_ax_mprage" in series_desc:
            info[t1w_axial].append(s.series_id)
        elif "sagittal_mp-rage" or "mprage_sag_iso" in series_desc:
            info[t1w_sagittal].append(s.series_id)
        elif "mprage_grappa2" or "mprage_bodycoil_ref" or "moco_nd_rms" or "moco_rms" or "passive_nd_rms" or "passive_rms" in series_desc:
            info[t1w_etc].append(s.series_id)
    ### /anat/t2w ###
        elif "t2_axial_space" in series_desc:
            info[t2w_axial].append(s.series_id)
        elif "t2_sag" in series_desc:
            info[t2w_sagittal].append(s.series_id)
        elif "t2w" or "180flip" or "cor_high_res" or "t2_nex_1" in series_desc:
            info[t2w_etc].append(s.series_id)
    ### /anat/flair ###
        elif "ax_flair" or "axial_flair" in series_desc:
            info[flair_axial].append(s.series_id)
        elif "flair_sag" in series_desc:
            info[flair_sagittal].append(s.series_id)
    ### /func/bold ###
        elif "ax_rsfmri" in series_desc:
            info[rsfmri_axial].append(s.series_id)
        elif "ax_rsfmri_a>>p" in series_desc:
            info[rs_ap_axial].append(s.series_id)
        elif "ax_rsfmri_p>>a" in series_desc:
            info[rs_pa_axial].append(s.series_id)
        elif "bold_resting_2x2x2" or "ep2d_bold_restingze6min" or "resting_bold_124" in series_desc:
            info[rs_etc].append(s.series_id)
        elif "rfmri_rest_pa" or "rfmri_rest_pa_sbref" or "long_rsfmri_p-a" in series_desc:
            info[rs_pa_etc].append(s.series_id)
    ### /dwi/dwi ###
        elif "axial_dti" in series_desc:
            info[dti_axial].append(s.series_id)
        elif "axial_dti_a>>p" in series_desc:
            info[dti_ap_axial].append(s.series_id)
        elif "dti_30dir_nodico_vox_1000" or "2x32" or "ep2d_dti_30dir_t" or "mddw_12" or "mddw_dti_axial_30_direction" in series_desc:
            info[dti_etc].append(s.series_id)
        elif "dti_30dir" in series_desc:
            info[dti_30dir].append(s.series_id)
        elif "dti_34dir" in series_desc:
            info[dti_34dir].append(s.series_id)
        elif "long_dti_p-a" or "no_aa_long_dti_p-a" in series_desc:
            info[dti_pa_etc].append(s.series_id)
    ### /localizer/localizer ###
        elif "localizer" or "calizer" in series_desc:
            info[localizer].append(s.series_id)
        #elif "ep2" in protocol:
            #info[ep2].append(s.series_id)
            #info[ep2_moco].append(s.series_id)
        else:
            print("Series not recognized!: ", protocol, s.dcm_dir_name)

    return info

def ReplaceSession(sesname):
    return sesname[:13].replace("-", "x").replace("_", "x")

[TinasheMTapera]

Could you also paste the output of running it? You could just do it at the command line and use --dry-run

[isabelannwingert]

@TinasheMTapera Here is the Output from Dry Run:

Gear Name: fw-heudiconv, Gear Version: 0.2.10_0.2.4
Gear starting...

/flywheel/v0/fw_heudiconv/backend_funcs/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: ==============: fw-heudiconv gear manager starting up :===============

/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py:6032: UserWarning: Client version 11.0.1 does not match server version 10.7.3. Please update your client version!
  warnings.warn('Client version {} does not match server version {}. Please update your client version!'.format(SDK_VERSION, release_version))
WARNING: Use "pip install flywheel-sdk~=10.7.3" to install a compatible version for this server
Traceback (most recent call last):
  File "./fw_heudiconv_run.py", line 49, in <module>
    subject_container = fw.get(session_container.parents['subject'])
  File "/usr/local/lib/python3.7/site-packages/flywheel/client.py", line 109, in get
    return self._fw.get(id, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py", line 6103, in get
    return self.get_container(id, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flywheel/flywheel.py", line 2143, in get_container
    return self.containers_api.get_container(container_id, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flywheel/api/containers_api.py", line 2486, in get_container
    (data) = self.get_container_with_http_info(container_id, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.7/site-packages/flywheel/api/containers_api.py", line 2521, in get_container_with_http_info
    raise ValueError("Missing the required parameter `container_id` when calling `get_container`")  # noqa: E501
ValueError: Missing the required parameter `container_id` when calling `get_container`

Gear completed unsuccessfully after running for 1.516s.
Uploading results...

[TinasheMTapera]

That's odd. Did you set any other parameters?

[isabelannwingert]

action	Curate
default_heuristic
do_whole_project	true
dry_run	false
extended_bids	true

[TinasheMTapera]

Hmm... Yeah the do whole project option may be buggy, can you try with one session?

[isabelannwingert]

Will do, have to wait til the gear finishes running 😦

[isabelannwingert]

Nope, does the same thing when applied to only one subject

[TinasheMTapera]

@isabelannwingert did you run this from the project level? I think that might be the problem.

@mattcieslak we haven't coded fw-heudiconv to run starting from the project, we hacked it to start from a session and have a flag for "do whole project" where it just starts at the session but goes to all the others. It looks like Flywheel has added functionality for gears starting from the project -- I think this used to just be project report but they now have an "Analysis" tab at the project level.

@isabelannwingert I can see that your gears started from the session always work, those started from the project don't. So right now your best solution is to start from one session and check "do whole project", we will start working on adding functionality to go from the full project.

That solves the first issue, could you paste the gear log of one of the successful ones so we can debug that?

[isabelannwingert]

@TinasheMTapera , sure, let me get that started. And yes, I ran this from the project level.

[TinasheMTapera]

I'm gonna close this issue, @isabelannwingert just remember the main issues were:

1. You can't use seqinfo data as part of the template string (yet)
2. Your boolean was incorrectly formed, just some python stuff:

In [43]: bool('string')
Out[43]: True

In [44]: bool('i' in 'string')
Out[44]: True

In [48]: bool('i' or 'x' in 'string')
Out[48]: True

In [49]: bool('y' or 'x' in 'string')
Out[49]: True

In [54]: bool('i' in 'string' or 'x' in 'string')
Out[54]: True

In [55]: bool('i' in 'string' and 'x' in 'string')
Out[55]: False