Closed jaredpz closed 4 years ago
Since you only have two runs could you try to hardcode run1 and run2 separately? i.e.
# field map
fmap_run1_ph = create_key(
'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-1_phasediff')
fmap_run1_mag = create_key(
'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-1_magnitude{item}')
fmap_run2_ph = create_key(
'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-2_phasediff')
fmap_run2_mag = create_key(
'sub-{subject}/{session}/fmap/sub-{subject}_{session}_run-2_magnitude{item}')
May narrow down the problem for me
The issue with that is that we haven't found an easy way to separate out run 1 and run 2 based on info contained in secinfo. See attached seqinfo .tsv file.
The only fields that meaningfully distinguish these runs are total_files_till_now
and date
, however to determine which one is first, we'd have to compare these values and then set the earlier one to be run-1 and later one to be run-2. I couldn't figure out an easy way to do this given that inside the for-loop within infotodict
you only have access to a single line of seqInfo
at a time.
This is why I was trying to use both item
and seqitem
in the keys, and in the case where it orders the B0maps in the correct order (i.e. case 2 from above), this heuristic works for organizing the data as we want (I think).
I'm confused as to why the sequence order listed here is changing, and why the only thing that changes is the ordering of the B0map phase images. I assume that the order of what gets printed to stdout
here is dictated by the order that the for loop inside infotodict
passes over the items in seqInfo
, which makes me think that for some reason seqInfo
is changing order from one call of fw-heudiconv-curate
to another. It looks like seqInfo is class OrderedDict, and I'm guessing this comes from a call to the python flywheel sdk? Is seqInfo
ordered based on the protocol_name
field? If it were ordered by dcm_dir_name
then it would already include the sequence number in the order and would be consistent, is there a way to do that?
Oops, seqInfo tsv attached here. TNI_SeqInfo.txt
That's interesting, I thought the scans were always ordered. @TinasheMTapera maybe we can sort them by acquisition time or total files til now before adding them to the seqinfo?
I can try that out and add it to the next patch, so it might take a couple of hours before I get that out to you @jaredpz
@jaredpz f4044db
I put in a small line that will force order acquisitions within the session to be ordered by date before generating seqinfos. Please use pip install fw-heudiconv==0.0.20
and let me know if they keep mixing up
This doesn't seem to have worked, both the B0maps and my rest scans are being inconsistently ordered, when I repeatedly run fw-heudiconv-curate
with the --dry_run
flag. It doesn't happen every time, and it's unpredictable when it will/wont happen See below:
First call
DEBUG:
B0map_13_ph.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG:
B0map_9_ph.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz
DEBUG:
restingBOLD_mb6_1200_14.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG:
restingBOLD_mb6_1200_10.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz
Second call
DEBUG:
B0map_13_ph.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-1_phasediff.nii.gz
DEBUG:
B0map_9_ph.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/fmap/sub-TNI.SC.1.009_ses-20170316113937_run-2_phasediff.nii.gz
DEBUG:
restingBOLD_mb6_1200_10.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-1_bold.nii.gz
DEBUG:
restingBOLD_mb6_1200_14.nii.gz
sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz -> sub-TNI.SC.1.009/ses-20170316113937/func/sub-TNI.SC.1.009_ses-20170316113937_task-rest_run-2_bold.nii.gz
The B0maps are ordered the same in each of these calls to fw-heudiconv-curate
, however they are in the wrong order each time (sequence 13, is being called run1, and sequence 8 is being called run2). However, the order of the rest scans is switched, and only in the second call are run1 and run2 labelled correctly. If I call fw-heudiconv-curate
10 time, I will probably get every possible permutation of the ordering for my B0map and rest sequences.
Okay, so actually this might be a bit more complicated/interesting.
When I look at the seqInfo that's written to stdout
before the heuristic file is applied, both calls to fw-heudiconv-curate
from above actually have the sequences listed in the right order, and identical to each other. One example in full below:
fw-heudiconv-curate --project TNI --subject TNI.SC.1.009 --session 20170316113937 --heuristic ./heuristic_TNI.py --dry_run --verbose
/Users/Jared/anaconda3/envs/flywheel/lib/python3.7/site-packages/fw_heudiconv/query.py:4: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution. We would be grateful for your help in improving them
from nibabel.nicom.dicomwrappers import wrapper_from_data
INFO: Querying Flywheel server...
DEBUG: Found project: TNI (5c4a19d6c2a4ad0037764016)
DEBUG: Found sessions:
20170316113937 (5d729ed10f98b7003970252c)
DEBUG: Found SeqInfos:
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b7005e7a2726)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(160, 160, 128, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed20f98b700356fc5d4)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b700376fe8ce)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 5, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed30f98b7003c70ca82)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)
AAHScout:
[TR=0.00315 TE=0.00137 shape=(162, 162, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'MPR', 'ND', 'NORM')] (5d729ed40f98b7003c70ca83)
Localizer_aligned:
[TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)
Localizer_aligned:
[TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)
Localizer_aligned:
[TR=0.104 TE=0.003 shape=(256, 256, 9, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b70043721ff1)
T1w_MPR:
[TR=2.4 TE=0.00222 shape=(320, 300, 208, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed40f98b700477567f4)
B0map:
[TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)
B0map:
[TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed50f98b7005e7a2729)
B0map:
[TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed50f98b70049778982)
restingBOLD_mb6_1200:
[TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed50f98b7003f70e645)
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)
Localizer:
[TR=0.04 TE=0.003 shape=(256, 256, 3, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND', 'NORM')] (5d729ed70f98b7004573e7f2)
B0map:
[TR=0.58 TE=0.00658 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)
B0map:
[TR=0.58 TE=0.00412 shape=(80, 80, 120, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'ND')] (5d729ed80f98b700477567f9)
B0map:
[TR=0.58 TE=0.00658 shape=(80, 80, 60, -1) image_type=('ORIGINAL', 'PRIMARY', 'P', 'ND')] (5d729ed80f98b70041717617)
restingBOLD_mb6_1200:
[TR=0.5 TE=0.025 shape=(448, 448, 1200, -1) image_type=('ORIGINAL', 'PRIMARY', 'M', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272d)
DSI_1.8mm_257dir_b5000_mb3:
[TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)
DSI_1.8mm_257dir_b5000_mb3:
[TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)
DSI_1.8mm_257dir_b5000_mb3:
[TR=4.3 TE=0.1022 shape=(1440, 1440, 258, -1) image_type=('ORIGINAL', 'PRIMARY', 'DIFFUSION', 'NONE', 'MB', 'ND', 'NORM', 'MOSAIC')] (5d729ed90f98b7005e7a272e)
By checking the series-ID, it's clear here that the sequences in seqInfo are ordered according to scan time, however when the heuristic is applied over seqinfo, it doesn't seem to be consistently applied in the same order, *and even when it is applied in a consistent order (i.e. with the B0 phase maps above) that order can be inconsistent with the order the sequences are listed in seqInfo *from the very same call of fw_heudiconv-curate
.
This makes me think possible the real culprit lies in my heuristic file. A copy of the exact one I used for these calls is attached. heuristic_TNI.txt
Okay so I seem to have fixed the problem by re-writing the heuristic file.
First, the heuristic file I uploaded with the last post actually has a typo in it, however that was not causing the problem (though I'm surprised it wasn't throwing an error inside create_key()
). A fixed copy of the original heuristic file that can recreate the problem listed above is attached here as heuristic_TNI.txt
I fixed the problem by implementing the get_both_series()
function you had in the example heuristic for my rest and b0-phase images, and for the b0 magnitude images I had to make a slight modification as follows:
def get_multi_b0mag(key1, key2, s):
if len(info[key1]) < 2:
info[key1].append(s.series_id)
else:
info[key2].append(s.series_id)
Working heuristic file is attached here as heuristic_TNI_v2.txt
heuristic_TNI_v2.txt
heuristic_TNI.txt
In my testing, I haven't found a case where this new heuristic mis-orders or misclassifies any of my data. That being said, without knowing what was causing the strange order issue above I'm not 100% confident this isn't going to make a mistake sometime. Empirically though this seems to work and I will try applying it to my dataset
@jaredpz thanks for bringing this to our attention, glad you got that worked out. I will dig a little deeper to find out what's going on.
In the meanwhile if you have the time, you could briefly describe your problem and solution and add it to the example heuristics repo by PR, in case other users have the same problem.
I'm running into problems with fw-heudiconv-curate where the order of the seqInfo OrderedDict seems to be inconsistent from one run to the next, leading to difficulty writing a working heuristic file. Heuristic file we're using is attached as text file.
Below is the
stdout
from two identical calls of fw-heudiconv-curate with the same heuristic file, which result in different organization/specification. The thing to pay attention to is in the list of sequences that BIDS heuristic was applied to, the order of the B0map phase imagesB0map_9_ph.nii.gz
andB0map_13_ph.nii.gz
are switched, and consequently the run assignment, which comes from theseqitem
variable, is also flipped.fw-heudiconv-curate --project TNI --subject TNI.SC.1.009 --session 20170316113937 --heuristic ./heuristic_TNI.py --dry_run --verbose
And the second time.
There are also two
restingBOLD
sequences but these seem to ordered and classified correctly every time. Ultimately, we want to specify that theB0map
run1 is intended forrestingBOLD
run1 and so on withB0map
run2, but right now we're not able to consistently get the B0maps classified in the proper order. Any idea on what's causing this?heuristic_TNI.txt