mrconvert fails

[martahedl]

Hi! I'm new to MRtirx3 and tried a couple of functions. Most work fine, but trying to convert a dicom-folder to .mif format fails badly (see attached picture). I tried to manipulate some options such as strides and grad-tables, but the results won't change. What can I do? Thanks in advance, Marlene.

[jdtournier]

I notice this is a Siemens mosaic image - any chance these data were anonymised or otherwise processed through a third-party DICOM handling utility? The mosaic format is not DICOM standard, and is specific to Siemens. The information required to read them correctly resides in specific private header entries in the DICOM headers, which are typically stripped out by anonymisation software and potentially other DICOM handling software. Without this information, there's not a lot that MRtrix3 can do other than make educated guesses about how the data were stored - and this means it'll likely get it wrong...

You can check whether your data are affected using the dcminfo command. If you can identify one DICOM file that is part of the problematic series, then run:

$ dcminfo -all -csa your_file

This will dump all the DICOM headers, including Siemens' private CSA attributes - which is where the information would be stored. If you see lines in the output that start with [CSA], then these headers are present, in which case there would indeed be a problem with MRtrix3 - but otherwise, the only thing I can realistically recommend is to try to get hold of the original unmodified DICOM data. Even if we could reformat the images correctly, the chances are you'd still be missing the DW encoding information, so there's no guarantee that your data would be usable anyway. See #1012 and #1061 for further discussions around this issue.

Can I also verify that you are running an up to date version of the software? I made changes for release candidate 2 that could have introduced issues in the handling of mosaic images - despite my extensive testing. It might be worth checkout out 3.0_RC1 to see whether it handles these data any differently...

However, if you do find the CSA entries in your data, let me know and we'll investigate further. More than likely, I'd need access to the data to figure out where the problem is...

[martahedl]

There are indeed many lines starting with csa (see below). However, I did not anonymize this specific dataset with a third party software. All I did was to import the dicoms to Horos, directly from the DVD from the scanner. Should I try to delete these entries?

 

[CSA] RealWorldValueMapping:

[CSA] PhaseContrastN4:

[CSA] MRVelocityEncoding:

[CSA] VelocityEncodingDirectionN4:

[CSA] ImageType4MF: ORIGINAL PRIMARY DIFFUSION NONE  

[CSA] VolumetricProperties4MF:

[CSA] MorphoQCThreshold:

[CSA] MorphoQCIndex:

[CSA] ImageHistory: ChannelMixing:ND=true_CMM=1_CDM=1 ACCAlgo:16 NormalizeAlgo:PreScan   

[CSA] MR_ASL:

[CSA] Distorcor_IntensityCorrection:

[DCM] 0029 1018 CS        2    24730   CSASeriesHeaderType                    [ MR ]

[DCM] 0029 1019 LO        8    24740   CSASeriesHeaderVersion                 [ 20170627 ]

[DCM] 0029 1020 OB   132484    24756   SiemensCSA2                            unknown data type

[CSA] UsedPatientWeight: 52           

[CSA] NumberOfPrescans: 0            

[CSA] TransmitterCalibration: 245.52833700     

[CSA] PhaseGradientAmplitude: 0.00000000     

[CSA] ReadoutGradientAmplitude: 0.00000000     

[CSA] SelectionGradientAmplitude: 0.00000000     

[CSA] GradientDelayTime: 35.00000000 35.00000000 30.00000000   

[CSA] RfWatchdogMask: 0            

[CSA] RfPowerErrorIndicator:

[CSA] SarWholeBody:

[CSA] Sed: 1000000.00000000 519.39208888 489.01886794   

[CSA] SequenceFileOwner: SIEMENS     

[CSA] Stim_mon_mode: 2            

[CSA] Operation_mode_flag: 0            

[CSA] dBdt_max: 0.00000000     

[CSA] t_puls_max: 0.00000000     

[CSA] dBdt_thresh: 0.00000000     

[CSA] dBdt_limit: 0.00000000     

[CSA] SW_korr_faktor: 1.00000000     

[CSA] Stim_max_online: 30.13399124 2.13742828 1.25491762   

[CSA] Stim_max_ges_norm_online: 0.71214372     

[CSA] Stim_lim: 42.70069885 23.97319984 36.48099899   

[CSA] Stim_faktor: 1.00000000     

[CSA] CoilForGradient: void     

[CSA] CoilForGradient2: AS82     

[CSA] CoilTuningReflection:

[CSA] CoilId: 255      0        0        0        0        4868     4867     4865     4866     0        0        

[CSA] MiscSequenceParam: 30       1        65       0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        524288   256      0        0        0        0        2        9444     0        0        0        0        0        0           

[CSA] MrProtocolVersion: 51130001     

[CSA] DataFileName:

[CSA] RepresentativeImage:

[CSA] PositivePCSDirections: +LPH     

[CSA] RelTablePosition: 0        0        8          

[CSA] ReadoutOS: 2.00000000     

[CSA] LongModelName: NUMARIS/4     

[CSA] SliceArrayConcatenations: 1            

[CSA] SliceResolution: 1.00000000     

[CSA] AbsTablePosition: -1247        

[CSA] AutoAlignMatrix:

[CSA] MeasurementIndex:

[CSA] CoilString: HE1-4     

[CSA] PATModeText: p2     

[CSA] PatReinPattern: 1;HFS;52.00;56.00;1;0;0;-80300943     

[CSA] ProtocolChangeHistory: 0            

[CSA] Isocentered: 1            

[CSA] MrPhoenixProtocol:  

 

 

Gesendet: Montag, 11. September 2017 um 00:36 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: martahedl marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

I notice this is a Siemens mosaic image - any chance these data were anonymised or otherwise processed through a third-party DICOM handling utility? The mosaic format is not DICOM standard, and is specific to Siemens. The information required to read them correctly resides in specific private header entries in the DICOM headers, which are typically stripped out by anonymisation software and potentially other DICOM handling software. Without this information, there's not a lot that MRtrix3 can do other than make educated guesses about how the data were stored - and this means it'll likely get it wrong...

You can check whether your data are affected using the dcminfo command. If you can identify one DICOM file that is part of the problematic series, then run:

$ dcminfo -all -csa your_file

This will dump all the DICOM headers, including Siemens' private CSA attributes - which is where the information would be stored. If you see lines in the output that start with [CSA], then these headers are present, in which case there would indeed be a problem with MRtrix3 - but otherwise, the only thing I can realistically recommend is to try to get hold of the original unmodified DICOM data. Even if we could reformat the images correctly, the chances are you'd still be missing the DW encoding information, so there's no guarantee that your data would be usable anyway. See #1012 and #1061 for further discussions around this issue.

Can I also verify that you are running an up to date version of the software? I made changes for release candidate 2 that could have introduced issues in the handling of mosaic images - despite my extensive testing. It might be worth checkout out 3.0_RC1 to see whether it handles these data any differently...

However, if you do find the CSA entries in your data, let me know and we'll investigate further. More than likely, I'd need access to the data to figure out where the problem is...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[martahedl]

Sorry, forgot to reply to your other question: Yes, I am running the latest version. I just downloaded MRtrix3 three days ago.  

Gesendet: Montag, 11. September 2017 um 00:36 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: martahedl marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

I notice this is a Siemens mosaic image - any chance these data were anonymised or otherwise processed through a third-party DICOM handling utility? The mosaic format is not DICOM standard, and is specific to Siemens. The information required to read them correctly resides in specific private header entries in the DICOM headers, which are typically stripped out by anonymisation software and potentially other DICOM handling software. Without this information, there's not a lot that MRtrix3 can do other than make educated guesses about how the data were stored - and this means it'll likely get it wrong...

You can check whether your data are affected using the dcminfo command. If you can identify one DICOM file that is part of the problematic series, then run:

$ dcminfo -all -csa your_file

This will dump all the DICOM headers, including Siemens' private CSA attributes - which is where the information would be stored. If you see lines in the output that start with [CSA], then these headers are present, in which case there would indeed be a problem with MRtrix3 - but otherwise, the only thing I can realistically recommend is to try to get hold of the original unmodified DICOM data. Even if we could reformat the images correctly, the chances are you'd still be missing the DW encoding information, so there's no guarantee that your data would be usable anyway. See #1012 and #1061 for further discussions around this issue.

Can I also verify that you are running an up to date version of the software? I made changes for release candidate 2 that could have introduced issues in the handling of mosaic images - despite my extensive testing. It might be worth checkout out 3.0_RC1 to see whether it handles these data any differently...

However, if you do find the CSA entries in your data, let me know and we'll investigate further. More than likely, I'd need access to the data to figure out where the problem is...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[jdtournier]

Right, so the CSA information is still there - look like we'll have to dig a little deeper...

Would you mind installing version 3.0_RC1 and trying again, just to rule out the changes in #1056 as the root cause of the problem? Instructions below:

$ git checkout 3.0_RC1
$ ./build

Assuming this still fails (which I expect it will), then there's a few questions that I can think of immediately. The first is that currently, a Siemens mosaic image without the relevant CSA information would be loaded as an actual mosaic, with a single slice, as reported in #1012. This is not what you're showing, so it looks like MRtrix3 is at least trying to reformat these data. But I'm not sure what happens if the relevant CSA information is not present, even if the entries where the information would normally be found is present - is the list of CSA above complete? I'd expect to see quite a bit more, and in particular information about the slice array (per-slice location, etc).

But in general, the simplest might be for you to send me the data so I can go properly investigate...

[martahedl]

Updating to 3.0_RC1 didn't help. I sent you a zip-folder of one of the datasets via wetransfer (to the email address jacques-donald.tournier@kcl.ac.uk, hope that's correct). Maybe you can figure out what's wrong.. Thanks so much in advance!  

Gesendet: Montag, 11. September 2017 um 13:36 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: "Marlene Tahedl" marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

Right, so the CSA information is still there - look like we'll have to dig a little deeper...

Would you mind installing version 3.0_RC1 and trying again, just to rule out the changes in #1056 as the root cause of the problem? Instructions below:

$ git checkout 3.0_RC1 $ ./build

Assuming this still fails (which I expect it will), then there's a few questions that I can think of immediately. The first is that currently, a Siemens mosaic image without the relevant CSA information would be loaded as an actual mosaic, with a single slice, as reported in #1012. This is not what you're showing, so it looks like MRtrix3 is at least trying to reformat these data. But I'm not sure what happens if the relevant CSA information is not present, even if the entries where the information would normally be found is present - is the list of CSA above complete? I'd expect to see quite a bit more, and in particular information about the slice array (per-slice location, etc).

But in general, the simplest might be for you to send me the data so I can go properly investigate...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[jdtournier]

OK, I have the data and I can reproduce the error. I'll look into it now...

[martahedl]

Allright, I'll wait patiently  

Gesendet: Dienstag, 12. September 2017 um 14:54 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: "Marlene Tahedl" marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

OK, I have the data and I can reproduce the error. I'll look into it now...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[jdtournier]

OK, it looks like the issue is that your data matrix was acquired as a 128×128 k-space matrix, zero-filled to 256×256, and stored as an 8×8 mosaic. But while I can easily get the 128×128 bit, it's remarkably hard to figure out where to look for the information that states that actually each slice is a 256×256 image... I'll dig around, see if I can find a fix.

In the meantime: I would strongly urge you to disable zero-filling in your protocol - it has a tendency of degrading low-SNR data (which is what we have in diffusion MRI) in non-obvious ways...

[martahedl]

Allright, I'll see to it that we change the protocol immediately. I hope, though, that you'll find the 256x256 information somewhere  

Gesendet: Dienstag, 12. September 2017 um 15:10 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: "Marlene Tahedl" marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

OK, it looks like the issue is that your data matrix was acquired as a 128×128 k-space matrix, zero-filled to 256×256, and stored as an 8×8 mosaic. But while I can easily get the 128×128 bit, it's remarkably hard to figure out where to look for the information that states that actually each slice is a 256×256 image... I'll dig around, see if I can find a fix.

In the meantime: I would strongly urge you to disable zero-filling in your protocol - it has a tendency of degrading low-SNR data (which is what we have in diffusion MRI) in non-obvious ways...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[jdtournier]

Right, I've just pushed a candidate fix in #1122, in the fix_handling_of_zero-filled_Siemens_mosaic branch. Feel free to give it a go - these instructions should do the trick:

$ git fetch
$ git checkout fix_handling_of_zero-filled_Siemens_mosaic
$ ./build

and see if that fixes it - certainly seems to work on my side.

When you're done, if you want to revert the code to its original state, this should do the trick:

$ git checkout master
$ ./build

[martahedl]

By visual inspection, mrconvert now works smoothly. Right now I'm testing some higher-level algorithms of the software on this dataset, I'm confident it'll work out. Thanks so much! However, I'll check the scanner for the zero filling. It seems none of us was aware that the protocol included zip in a first place. Are you positive this was the source of the error?  

Gesendet: Dienstag, 12. September 2017 um 22:41 Uhr Von: "J-Donald Tournier" notifications@github.com An: MRtrix3/mrtrix3 mrtrix3@noreply.github.com Cc: "Marlene Tahedl" marlene.tahedl@gmx.de, Author author@noreply.github.com Betreff: Re: [MRtrix3/mrtrix3] mrconvert fails (#1118)

Right, I've just pushed a candidate fix in #1122, in the fix_handling_of_zero-filled_Siemens_mosaic branch. Feel free to give it a go - these instructions should do the trick:

$ git fetch $ git checkout fix_handling_of_zero-filled_Siemens_mosaic $ ./build

and see if that fixes it - certainly seems to work on my side.

When you're done, if you want to revert the code to its original state, this should do the trick:

$ git checkout master $ ./build

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

 

[jdtournier]

By visual inspection, mrconvert now works smoothly. Right now I'm testing some higher-level algorithms of the software on this dataset, I'm confident it'll work out. Thanks so much!

Good to hear!

However, I'll check the scanner for the zero filling. It seems none of us was aware that the protocol included zip in a first place.

Yes, zero-filling can be a pain to disable. Thankfully it's a lot easier on Siemens scanners than on GE or Philips. I can't remember what the setting was exactly, but it should be fairly self-evident once you come across it.

Are you positive this was the source of the error?

Yes, but I'd phrase it slightly differently :wink: This exposed a bug in MRtrix3's handling of mosaics when combined with zero-filling - ZIP is not the source of the problem as such...

However, there's many reasons why zero-filling is a problem in DWI (beyond exposing bugs in our software). When combined with magnitude reconstruction on low SNR data, it introduces strange structures in the data due to the strong Gibbs ringing pushing the signal to go 'negative' (i.e. opposite phase in the complex domain) but being reconstructed positive after the magnitude transform. These artefacts will then propagate through to the reconstructed maps. See image below for example (raw b=3000 DWI on a GE scanner - acquired a very long time ago...):

It will also mean your data will not be suitable for Gibbs ringing removal (via the new mrdegibbs command), and may also be problematic for MP-PCA denoising (via the dwidenoise command) - although that last one is not obvious, I'd need to check.

---

And to finish off, here's the evidence that zero-filling was used, from your data (dcminfo -a -csa IM-0001-0001.dcm | less):

The k-space data matrix was acquired at 128×128, as per the official DICOM entry:

[DCM] 0018 1310 US        8     8394   AcquisitionMatrix                      [ 128 0 0 128 ]

the Siemens non-standard Custom Shadow Attributes (CSA) entry:

[CSA] AcquisitionMatrixText: 128p*128     

and the even less standard ASCONV entry within one of the CSA entries:

sKSpace.lBaseResolution  =      128
sKSpace.lPhaseEncodingLines      =      128

But the image is clearly 256×256, as you can clearly verify in the correctly reconstructed images, and as hinted at by the combination of the in-place voxel size (standard DICOM entry):

[DCM] 0028 0030 DS        4     9758   PixelSpacing                           [ 1 1 ]

and the FoV (stored in the non-standard ASCONV entry relating to the mosaic information):

sAdjData.sAdjVolume.dPhaseFOV    =      256.0
sAdjData.sAdjVolume.dReadoutFOV  =      256.0

A slightly oblique way to show that the reconstructed matrix must be 256×256...

The only other reference to the correct reconstructed matrix size is right near the end, in a Siemens-only (private DICOM tags) section that presumably contains text entries to be displayed on the UI (presumably you can verify this by displaying the images on the scanner, or using their own DICOM viewer):

[DCM] 0051 100B LO       10   156736   unknown                                [ 256p*256 I ]

presumably the I in that entry indicates interpolation? Just a guess though...

[martahedl]

Wow, that clearly taught me a lesson on DWI acquisition and data handling - I really appreciate it! It's so much fun to discover these things 👍 now comes the handling of scanner guts... :D Denoising seems to work fine, but then again I can't really tell since I have no non-zero-filled images to do the comparison.

[martahedl]

sorry, you will need of course the residual map to judge the denoising quality - maybe there is a little anatomical structure recognizable around the CSF area?

[jdtournier]

Denoising seems to work fine

Yes, I don't expect the command itself to have any trouble processing these data - it's more whether the results are valid. And given your noise map, it looks OK to me... Although I am surprised to find higher noise outside the brain - typically we see the opposite. This might indicate that it's not denoising as much as it otherwise would. But it also shouldn't harm the data, so that might be fine.

[martahedl]

All right, will do. Needless to say you were right (also with interpolation), we just checked the scanner. Allow me one last question: Above, you posted you could easily get the 128x128 bit. Does that mean you can access the original data in the 128x128 k-space? Where would I find that? If possible, I would like to work with that data instead of the zero-filled, but the original raw data has long been cleared from the scanner...

[jdtournier]

Above, you posted you could easily get the 128x128 bit. Does that mean you can access the original data in the 128x128 k-space?

No! That would indeed be pretty useful... What I meant is I can easily and robustly ascertain that the data were acquired using a 128×128 k-space matrix - that's in a standard DICOM tag (as per my previous post). But DICOM does not store the original k-space data...

[jdtournier]

@martahedl: these changes have now been merged to master - you can switch back to that branch now:

git checkout master
git pull
./build

I'll close this issue now. Thanks for reporting the problem!