search

PennLINC / fw-heudiconv

Heuristic-based Data Curation on Flywheel
BSD 3-Clause "New" or "Revised" License
6 stars 11 forks source link

fw-heudiconv-curate seqInfo order inconsistency #43

Closed jaredpz closed 4 years ago

jaredpz commented 4 years ago

I'm running into problems with fw-heudiconv-curate where the order of the seqInfo OrderedDict seems to be inconsistent from one run to the next, leading to difficulty writing a working heuristic file. Heuristic file we're using is attached as text file.

Below is the stdout from two identical calls of fw-heudiconv-curate with the same heuristic file, which result in different organization/specification. The thing to pay attention to is in the list of sequences that BIDS heuristic was applied to, the order of the B0map phase images B0map_9_ph.nii.gz and B0map_13_ph.nii.gz are switched, and consequently the run assignment, which comes from the seqitem variable, is also flipped.

fw-heudiconv-curate --project TNI --subject TNI.SC.1.009 --session 20170316113937 --heuristic ./heuristic_TNI.py --dry_run --verbose

/Users/Jared/anaconda3/envs/flywheel/lib/python3.7/site-packages/fw_heudiconv/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: Querying Flywheel server...
DEBUG: Found project: TNI (5c4a19d6c2a4ad0037764016)
DEBUG: Found sessions:
    20170316113937 (5d729ed10f98b7003970252c)
DEBUG: Found SeqInfos:
AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(160, 160, 128, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b700356fc5d4)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed50f98b70049778982)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed80f98b70041717617)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    T1w_MPR: 
        [TR=2.4 TE=0.00222 shape=(320, 300, 208, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b700477567f4)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed50f98b7003f70e645)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272d)

INFO: Loading heuristic file...
INFO: Applying heuristic to query results...
DEBUG: 
T1w_MPR_7.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_T1w.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/anat/sub-TNI.SC.1.009_ses-20170316113937_T1w.nii.gz
DEBUG: 
B0map_13_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG: 
B0map_9_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz
DEBUG: 
B0map_12_e1.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude1.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude1.nii.gz
DEBUG: 
B0map_12_e2.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude2.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude2.nii.gz
DEBUG: 
B0map_8_e1.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude1.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude1.nii.gz
DEBUG: 
B0map_8_e2.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude2.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude2.nii.gz
DEBUG: 
restingBOLD_mb6_1200_10.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG: 
restingBOLD_mb6_1200_14.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.bval
    sub-TNI.SC.1.009_ses-20170316113937_dwi.bval -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.bval
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.bvec
    sub-TNI.SC.1.009_ses-20170316113937_dwi.bvec -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.bvec
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_dwi.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.nii.gz

And the second time.

/Users/Jared/anaconda3/envs/flywheel/lib/python3.7/site-packages/fw_heudiconv/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: Querying Flywheel server...
DEBUG: Found project: TNI (5c4a19d6c2a4ad0037764016)
DEBUG: Found sessions:
    20170316113937 (5d729ed10f98b7003970252c)
DEBUG: Found SeqInfos:
AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(160, 160, 128, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b700356fc5d4)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed50f98b70049778982)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed80f98b70041717617)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    T1w_MPR: 
        [TR=2.4 TE=0.00222 shape=(320, 300, 208, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b700477567f4)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed50f98b7003f70e645)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272d)

INFO: Loading heuristic file...
INFO: Applying heuristic to query results...
DEBUG: 
T1w_MPR_7.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_T1w.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/anat/sub-TNI.SC.1.009_ses-20170316113937_T1w.nii.gz
DEBUG: 
B0map_9_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG: 
B0map_13_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz
DEBUG: 
B0map_8_e1.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude1.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude1.nii.gz
DEBUG: 
B0map_8_e2.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude2.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_magnitude2.nii.gz
DEBUG: 
B0map_12_e1.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude1.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude1.nii.gz
DEBUG: 
B0map_12_e2.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude2.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_magnitude2.nii.gz
DEBUG: 
restingBOLD_mb6_1200_10.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG: 
restingBOLD_mb6_1200_14.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.bval
    sub-TNI.SC.1.009_ses-20170316113937_dwi.bval -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.bval
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.bvec
    sub-TNI.SC.1.009_ses-20170316113937_dwi.bvec -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.bvec
DEBUG: 
DSI_1.8mm_257dir_b5000_mb3_15.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_dwi.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/dwi/sub-TNI.SC.1.009_ses-20170316113937_dwi.nii.gz

There are also two restingBOLD sequences but these seem to ordered and classified correctly every time. Ultimately, we want to specify that the B0map run1 is intended for restingBOLD run1 and so on with B0map run2, but right now we're not able to consistently get the B0maps classified in the proper order. Any idea on what's causing this?

heuristic_TNI.txt

TinasheMTapera commented 4 years ago

Since you only have two runs could you try to hardcode run1 and run2 separately? i.e.

# field map
fmap_run1_ph = create_key(
    'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-1_phasediff')
fmap_run1_mag = create_key(
    'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-1_magnitude{item}')

fmap_run2_ph = create_key(
    'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-2_phasediff')
fmap_run2_mag = create_key(
    'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-2_magnitude{item}')

May narrow down the problem for me

jaredpz commented 4 years ago

The issue with that is that we haven't found an easy way to separate out run 1 and run 2 based on info contained in secinfo. See attached seqinfo .tsv file.

The only fields that meaningfully distinguish these runs are total_files_till_now and date, however to determine which one is first, we'd have to compare these values and then set the earlier one to be run-1 and later one to be run-2. I couldn't figure out an easy way to do this given that inside the for-loop within infotodict you only have access to a single line of seqInfo at a time.

This is why I was trying to use both item and seqitem in the keys, and in the case where it orders the B0maps in the correct order (i.e. case 2 from above), this heuristic works for organizing the data as we want (I think).

I'm confused as to why the sequence order listed here is changing, and why the only thing that changes is the ordering of the B0map phase images. I assume that the order of what gets printed to stdout here is dictated by the order that the for loop inside infotodict passes over the items in seqInfo, which makes me think that for some reason seqInfo is changing order from one call of fw-heudiconv-curate to another. It looks like seqInfo is class OrderedDict, and I'm guessing this comes from a call to the python flywheel sdk? Is seqInfo ordered based on the protocol_name field? If it were ordered by dcm_dir_name then it would already include the sequence number in the order and would be consistent, is there a way to do that?

jaredpz commented 4 years ago

Oops, seqInfo tsv attached here. TNI_SeqInfo.txt

mattcieslak commented 4 years ago

That's interesting, I thought the scans were always ordered. @TinasheMTapera maybe we can sort them by acquisition time or total files til now before adding them to the seqinfo?

TinasheMTapera commented 4 years ago

I can try that out and add it to the next patch, so it might take a couple of hours before I get that out to you @jaredpz

TinasheMTapera commented 4 years ago

@jaredpz f4044db I put in a small line that will force order acquisitions within the session to be ordered by date before generating seqinfos. Please use pip install fw-heudiconv==0.0.20 and let me know if they keep mixing up

jaredpz commented 4 years ago

This doesn't seem to have worked, both the B0maps and my rest scans are being inconsistently ordered, when I repeatedly run fw-heudiconv-curate with the --dry_run flag. It doesn't happen every time, and it's unpredictable when it will/wont happen See below:

First call

DEBUG: 
B0map_13_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG: 
B0map_9_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz

DEBUG: 
restingBOLD_mb6_1200_14.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG: 
restingBOLD_mb6_1200_10.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz

Second call

DEBUG: 
B0map_13_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG: 
B0map_9_ph.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz

DEBUG: 
restingBOLD_mb6_1200_10.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG: 
restingBOLD_mb6_1200_14.nii.gz
    sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz

The B0maps are ordered the same in each of these calls to fw-heudiconv-curate, however they are in the wrong order each time (sequence 13, is being called run1, and sequence 8 is being called run2). However, the order of the rest scans is switched, and only in the second call are run1 and run2 labelled correctly. If I call fw-heudiconv-curate 10 time, I will probably get every possible permutation of the ordering for my B0map and rest sequences.

jaredpz commented 4 years ago

Okay, so actually this might be a bit more complicated/interesting.

When I look at the seqInfo that's written to stdout before the heuristic file is applied, both calls to fw-heudiconv-curate from above actually have the sequences listed in the right order, and identical to each other. One example in full below:

fw-heudiconv-curate --project TNI --subject TNI.SC.1.009 --session 20170316113937 --heuristic ./heuristic_TNI.py --dry_run --verbose
/Users/Jared/anaconda3/envs/flywheel/lib/python3.7/site-packages/fw_heudiconv/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: Querying Flywheel server...
DEBUG: Found project: TNI (5c4a19d6c2a4ad0037764016)
DEBUG: Found sessions:
    20170316113937 (5d729ed10f98b7003970252c)
DEBUG: Found SeqInfos:
Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(160, 160, 128, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b700356fc5d4)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    AAHScout: 
        [TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    Localizer_aligned: 
        [TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)

    T1w_MPR: 
        [TR=2.4 TE=0.00222 shape=(320, 300, 208, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b700477567f4)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed50f98b70049778982)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed50f98b7003f70e645)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    Localizer: 
        [TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)

    B0map: 
        [TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed80f98b70041717617)

    restingBOLD_mb6_1200: 
        [TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272d)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

    DSI_1.8mm_257dir_b5000_mb3: 
        [TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)

By checking the series-ID, it's clear here that the sequences in seqInfo are ordered according to scan time, however when the heuristic is applied over seqinfo, it doesn't seem to be consistently applied in the same order, *and even when it is applied in a consistent order (i.e. with the B0 phase maps above) that order can be inconsistent with the order the sequences are listed in seqInfo *from the very same call of fw_heudiconv-curate.

This makes me think possible the real culprit lies in my heuristic file. A copy of the exact one I used for these calls is attached. heuristic_TNI.txt

jaredpz commented 4 years ago

Okay so I seem to have fixed the problem by re-writing the heuristic file.

First, the heuristic file I uploaded with the last post actually has a typo in it, however that was not causing the problem (though I'm surprised it wasn't throwing an error inside create_key()). A fixed copy of the original heuristic file that can recreate the problem listed above is attached here as heuristic_TNI.txt

I fixed the problem by implementing the get_both_series() function you had in the example heuristic for my rest and b0-phase images, and for the b0 magnitude images I had to make a slight modification as follows:

def get_multi_b0mag(key1, key2, s):
         if len(info[key1]) < 2:
             info[key1].append(s.series_id)
         else:
             info[key2].append(s.series_id)

Working heuristic file is attached here as heuristic_TNI_v2.txt heuristic_TNI_v2.txt heuristic_TNI.txt

In my testing, I haven't found a case where this new heuristic mis-orders or misclassifies any of my data. That being said, without knowing what was causing the strange order issue above I'm not 100% confident this isn't going to make a mistake sometime. Empirically though this seems to work and I will try applying it to my dataset

TinasheMTapera commented 4 years ago

@jaredpz thanks for bringing this to our attention, glad you got that worked out. I will dig a little deeper to find out what's going on.

In the meanwhile if you have the time, you could briefly describe your problem and solution and add it to the example heuristics repo by PR, in case other users have the same problem.