DICOMDIR-data, missing temp folders / provenance store

[karl-koschutnig]

Hi experts,

I would really appreceate your help with the following issue: I am trying to convert Siemens raw data in the "Dicomdir organization" to BIDS (obviously). When starting the bids mapper, several temp folder are automatically generated. For example: /var/folders/5m/fn9m7txd3nx0m6c8r6r1h39m0000gn/T/tmpv2h7kv8k/Users/koschutnigk/MRILab/bidscoiner/raw/sub-000/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/099-PhoenixZIPReport/35743028

Problem is that when the GUI is opening, all of theses folders are gone! For example /var/folders/5m/fn9m7txd3nx0m6c8r6r1h39m0000gn/T/tmpv2h7kv8k/Users/koschutnigk/MRILab/bidscoiner/raw/sub-000/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/099-PhoenixZIPReport/35743028 not found

Then, when trying to set things up in the mapper-GUI it says: "Cannot reliably change the datatype and/or suffix because the source file '/var/folders/5m/fn9m7txd3nx0m6c8r6r1h39m0000gn/T/tmpv2h7kv8k/Users/koschutnigk/MRILab/bidscoiner/raw/sub-000/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/006-fmap_acq-FRest_dir-PA/35748748' can no longer be found.

Please restore the source data or use the bidsmapper -s option to solve this issue. Resetting the run-item now..."

First I thought it is a MacOS problem but then I had the same issue under Ubuntu. I really would like to stick with the "DICOMdir organization" format because we acquire MPM data. For the MPM-data we have different echo-times. For these I need separate folders so that the bids coiner sort them correctly.

Thanks for your help

Cheers Karl

[marcelzwiers]

Hi Karl,

I personally don't work with DICOMDIR, so I did not fully test your use case. What happens is that I reorder your DICOMDIR data in a temporary directory into the structure that BIDScoin can handle directly (i.e. into the structure we use in our centre). Because it's temporary data, it is deleted and no longer available by the editor. This can pose a problem when you change the datatype into a datatype that is identified with different DICOM fields. Then the bidseditor needs to access the DICOM data to read the new fields. As printed in the warning you saw, a work-around is to store a DICOM sample of each datatype into the bids folder, and this is what the -s option is supposed to do. Have you tried this?

[marcelzwiers]

p.s. 1) If you are re-running the bidsmapper with the -s option, then either start with a new bidsmap / bidsfolder, or also use the -f option (which deletes your previous bidsmap). 2) Another solution is to permanently reorder your DICOMDIR data yourself using the dicomsort utility (and only then run bidsmapper -f). But that is probably not preferable?

[marcelzwiers]

Maybe I will implement more direct support for DICOMDIR in a future release, but so far I haven't seen people using it

[marcelzwiers]

Question, do you have a DICOMDIR per session, per subject, or perhaps one DICOMDIR for your whole study?

[karl-koschutnig]

Hi Marcel, and huge thanks for your fast and detailed answers!! I tried the "-s" option - didn't work. I also tried the dicomsort command (and other sorting tools). For me the main problem is that I need separat folders for each echo Time for our MPM data (at the moment I get 6 dicoms in one folder - it should be 6 folders with 1 dcm each). Dicomsort crashes when specifying EchoTime (because in some cases (some sequences) there is no echo time). It would be perfectly fine for me to use discomfort, if there is a "echo time" workaround!

We have one DICOMDIR per subject.

[marcelzwiers]

EchoTime, are you sure? Because if I look back at the code I don't see EchoTime being used at all...

[marcelzwiers]

Or did you supply a folderscheme yourself?

[marcelzwiers]

If you got this warning:

LOGGER.warning(f"Missing '{field}' DICOM field specified in the '{scheme}' naming scheme, cannot find a safe name for: {dicomfile}\n")

Then I can fix it easily. But you said it crashed?

[karl-koschutnig]

That's the one:

Missing 'EchoTime' DICOM field specified in the '{SeriesNumber:03d}-{SeriesDescription}-{EchoTime:05d}' naming scheme, cannot find a safe name for: /Volumes/Thunder/CP02/converted/sub-000/ses-1/26_fmap_acq-b1TFL_dir-AP_UNKNOWN/image0001.dcm

[marcelzwiers]

The dicomsort in the latest code now has a --force option to sort dicoms, even in the absence of DICOM fields listed in your folder/name scheme

[marcelzwiers]

But I'd like to also fix the -s solution, as it was not working for you. What happened there, did you get any warnings or errors?

[marcelzwiers]

The latest github code should now support the bidsmapper -s once again, so I would be grateful if you could test that :-)

[karl-koschutnig]

Hi and again thanks for your great work!! The "-s" works now perfectly. Within the code/bidscoin/provenance/.. all data can be found - awesome.

Unfortunatelly, the dicomsort didn't run successfully. Here is the command and the output (maybe I missed something?):

dicomsort /Volumes/Thunder/CP02/raw/sub-000 -f {SeriesNumber:03d}-{SeriesDescription}-{EchoTime:05d} -n {AcquisitionNumber:05d}_{InstanceNumber:05d}.dcm --force INFO | >> Sorting: /Volumes/Thunder/CP02/raw/sub-000/124_BP_CP02_1_1_999/01-Bio Psychologie 2022 (722 files) Traceback (most recent call last): File "/Users/karlkoschutnig/opt/miniconda3/bin/dicomsort", line 8, in sys.exit(main()) File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/dicomsort.py", line 233, in main sortsessions(sourcefolder = args.dicomsource, File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/dicomsort.py", line 179, in sortsessions sortsession(sessionfolder, dicomfiles, folderscheme, namescheme, force, dryrun) File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/dicomsort.py", line 112, in sortsession subfolder = construct_name(folderscheme, dicomfile, force) File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/dicomsort.py", line 47, in construct_name return scheme.format(**schemevalues) if schemevalues else '' ValueError: Unknown format code 'd' for object of type 'str'

[karl-koschutnig]

I just recognized that the output of the plain dicomsort doesn't result in the same output as the bidsmapper. The MPM-data (in our case with 6 echo times) has 6 dicoms after dicomsort, but only one dicom (the last one) is stored in the bidsmapper (provenance). See Folder 033 in the screenshot:

[marcelzwiers]

I'll look into that error, but as for the provenance store, I store only one data sample per datatype, not all the raw data

[marcelzwiers]

ValueError: Unknown format code 'd' for object of type 'str' suggests that you are getting a string from the dicom field, but in your naming scheme you expect it to be an integer? What was your folderscheme?

[karl-koschutnig]

... in most cases the TE is not stored in the DICOM. With other dicom-sorter the folder results in "Unknown" (For example: 06_fmap_acq-FRest_dir-PA_UNKNOWN). So, I guess they put a string in the name, if there is no number ?! However, I just figured out that the TE is also not present in the MPM-data (which really doesn't make any sense). I think this is related to the XA31 Dicom changes (we had so many problems with the new XA).

Sorry, for all that, I have to find another way to sort this kind of data.

One last question: When I use the bidseditor for the MPM-data, the output is : sub-000_acq-T1we1a_run-1_flip-1_mt-off_MPM.nii.gz /json sub-000_acq-T1we1b_run-1_flip-1_mt-off_MPM.nii.gz / json

Is there a was to bring the "e1a" "e1b" etc. to echo-1, echo-2 sub-000_acq-T1w_run-1flip-1echo-1_mt-off_MPM.nii.gz /json sub-000_acq-T1w_run-1flip-1echo-2_mt-off_MPM.nii.gz / json

I know that the "e1a" comes from dcm2niix, but maybe there is a way to map this in a different way?

Again, thanks for all your work!!

[marcelzwiers]

That e1a suffix is added by dcm2niix and I make an effort in my code to move that suffix into the acq-label if there is no echo key-value pair allowed for that BIDS datatype. That is, I do that if 'echo' in run['bids'] and postfix.startswith('e'):. In your case your postfix also has a a, b, etc (probably due to having multiple images in one seriesfolder), maybe that messes things up

[marcelzwiers]

Did you get: ERROR: Unexpected postix 'e1a' found in {dcm2niixfile}?

[karl-koschutnig]

Yes: 2022-11-25 09:19:32 - ERROR | Unexpected postix 'e1a' found in /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-T1w_run-1_flip-1_mt-off_MPM_e1a.nii.gz

[marcelzwiers]

If I'm right, then that additional a in e1a will disappear with proper DICOM sorting (i.e. such that dcm2niix produces one output image per DICOM folder) and then the e1 will automatically be moved to the echo label

[marcelzwiers]

I can make an additional catch for this use case

[karl-koschutnig]

Yeah, that's what I thought in the first place (to make different folders for different EchoTimes) - which is obviously not that easy. An additional catch would be great!

[marcelzwiers]

I added 3 lines of code that should do the trick (hopefully)

[marcelzwiers]

p.s. You said that TE is not present in the DICOM header, but dcm2niix is recognizing the different echos, so I guess the info must be there (somewhere). In the bidseditor, you can double click IMA/DICOM files to inspect their header. Also, you can test things out by entering (single bracket) dynamic values in the edit window (e.g. <Echo Time> or <EchoTime>)

[marcelzwiers]

[karl-koschutnig]

Thanks for the update. That's a great way to specify the echos!

However, I am not sure if there is the EchoTime in the DicomTags. I looked them up in Soros and the entries that I found with "echo" are: EchoPulseSequence, EchoPlanarPulseSeqeuence .

Then I used dcminfo (mrtrix) and found a lot of EchoTime entries, like: [DCM] 0018 9114 SQ 0 463938 > MREchoSequence
[DCM] FFFE E000 UN 0 463950 - Item
[DCM] 0018 9082 FD 8 463958 EffectiveEchoTime 4.92 [DCM] FFFE E00D UN 0 463974 ItemDelimitationItem unknown data type [DCM] FFFE E0DD UN 0 463982 SequenceDelimitationItem unknown data type

somewhat nested.

If you are interested, I've attached the complete output of the dcminfo-command MPM_tags.txt

[karl-koschutnig]

Please find attached the latest logfile (with your latest updates). It seems that the first echo is set correctly but all others are not captured. bidscoin_latest.log

[marcelzwiers]

Mhhh, are you sure you have the latest code? Sometimes pip install --upgrade doesn't upgrade if the package version is the same (which is the case here). Can you verify that you the following lines (329-331) in your dcm2niix2bids.py plugin (you can find them with bidscoin -p)?

                        elif echonr[0:-1].isnumeric():
                            newbidsname = bids.insert_bidskeyval(newbidsname, 'echo', echonr[0:-1].lstrip('0'), bidsignore) # Strip of the 'a', 'b', etc from `e1a`, `e1b`, etc
                            newbidsname = bids.get_bidsvalue(newbidsname, 'dummy', echonr[-1])                  # Append the 'a' to the acq-label

[karl-koschutnig]

You are right - they are not the same (see below): sorry for that. What is the best way to upgrade bidscoin? I did: pip install --upgrade git+https://github.com/Donders-Institute/bidscoin

else: LOGGER.error(f"Unexpected postix '{postfix}' found in {dcm2niixfile}") newbidsname = bids.get_bidsvalue(newbidsname, 'dummy', postfix) # Append the unknown postfix to the acq-label

[marcelzwiers]

You can uninstall and then install or do:

pip install --force-reinstall --no-deps git+https://github.com/Donders-Institute/bidscoin

[karl-koschutnig]

Thanks - I managed to update to the latest version!

What I get know is this

... echo is not updated (stays at 1) but letters are added at the acq.

From the terminal: VERBOSE | Found dcm2niix ['e1'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_echo-1_flip-25_mt-off_MPM.nii.gz VERBOSE | Found dcm2niix ['e1a'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1a.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1wa_run-1_echo-1_flip-25_mt-off_MPM.nii.gz VERBOSE | Found dcm2niix ['e1b'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1b.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1wb_run-1_echo-1_flip-25_mt-off_MPM.nii.gz VERBOSE | Found dcm2niix ['e1c'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1c.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1wc_run-1_echo-1_flip-25_mt-off_MPM.nii.gz VERBOSE | Found dcm2niix ['e1d'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1d.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1wd_run-1_echo-1_flip-25_mt-off_MPM.nii.gz VERBOSE | Found dcm2niix ['e1e'] postfixes, renaming
/Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1w_run-1_flip-25_mt-off_MPM_e1e.nii.gz -> /Volumes/Thunder/CP02/sub-000/anat/sub-000_acq-MPMT1we_run-1_echo-1_flip-25_mt-off_MPM.nii.gz

[marcelzwiers]

Well, I guess that's the best dcm2niix can do, given that there is no echo time, i.e. labeling them all as echo-1 and then put a a, b, c, etc suffix to avoid overwriting the files (probably the order is random). But... you can perhaps be smarter than dcm2niix and use <<EffectiveEchoTime>> somewhere in your bidsname. For instance you can give this a try (don't know if pydicom can find this tag):

(don't put it in echo for now, because with my latest patch that will be replaced with acq-MPMT1wa_*_echo-1 )

[marcelzwiers]

Mhhh, interesting, I see in your output that dcm2niix says:

Slices not stacked: echo varies (TE 9.84, 17.22; echo 1, 1)

But e.g. 9.84 is not present in the dicom dump you sent...

[karl-koschutnig]

??? I am lost. It has to be somewhere in the enhanced-Dicom eco-system. I'll try out the EffectiveEchoTime.

Again, thank you so much - great effort!!

[karl-koschutnig]

Finally, at least I managed to get the proper folder structure! I mean I tried what you already had prepared ;-)

dicomsort /Volumes/Thunder/CP02/raw3/CP02Rest -f {SeriesNumber:03d}-{SeriesDescription}-{EffectiveEchoTime} -n {AcquisitionNumber}{InstanceNumber:05d}.dcm --force

From the terminal: INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-17.22 INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-14.76 INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-12.3 INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-9.84 INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-7.38 INFO | Creating: /Volumes/Thunder/CP02/raw3/CP02_Rest/124_BP_CP02_1_1_999/01-Bio Psychologie 2022/045-anat-MTw_acq-S-4.92

[marcelzwiers]

Great! I expect that the next step is that you have to add the EffectiveEchoTime attribute to the template bidsmap (because thus far bidscoin has been blind about this tag):

  - provenance:
    attributes:
      <<: *anat_dicomattr
      ProtocolName: (?i).*MPM.*
      EffectiveEchoTime:
    bids: &anat_dicoment_vfamt 
      acq: <SeriesDescription>
      ce:
      rec:
      run: <<1>>
      echo:
      flip:
      mt:
      part: ['', mag, phase, real, imag, 0]
      suffix: MPM

[marcelzwiers]

Btw, the MPM json-file that is produced by dcm2niix, does that contain any info on the echo time? If not, I can add EffectiveEchoTime: <<EffectiveEchoTime>> to the meta data

[karl-koschutnig]

As a fact it is there - it somehow reincarnated?! "EchoTime": 0.01722

[marcelzwiers]

If you still have issues or if there's something more I can do, then don't hesitate to reopen this issue