ENH NIfTI: Use NIFTI_INTENT_RGB_VECTOR when appropriate

Some MRItrix Track Density Imaging (TDI) images are saved as 3-volume float-32 NIfTI images. The intention is for these volumes to be displayed as red, green, and blue components. Unfortunately, the intention field of the NIfTI header is not set, so most tools assume they are 3-volume scalar images. See here for an example.

The solution is simple, the intent_code of the NIfTI header should be set to 2003 (NIFTI_INTENT_RGB_VECTOR):

To signify that the vector value at each location is an RGB triplet,
     of whatever type.

This will allow tools to automatically recognize the intention for the images. The sample dataset reports the image was generated by MRtrix version: 3.0_RC3.

Thanks for the suggestion.

The fact that MRtrix3 supports a wide range of image formats for all commands makes this slightly trickier. I.e. tckmap can't simply "set the intent code", because it's not itself responsible for NIfTI export, that all goes through very generic MRtrix3 backend code.

What I'm thinking here is doing something akin to my solution for the MGH format. There, I use dedicated key-value pairs prefixed with "MGH_*" at both read and write, so that e.g. mrconvert from MGH to MGH preserves sidecar information that is specific to that format despite having passed through the intermediate internal MRtrix3 representation (which is just std::map<std::string, std::string>). So translating that technique to this case, on NIfTI read & write, key-value "NIFTI_INTENT" could be read & written in the relevant file handling; tckmap could then simply write this field to the output image header if -dec is used, and the NIfTI file handling will pick it up.

Just to add my 2¢ to this issue: what @Lestropie said addresses how this might be implemented, and this is indeed how we'd have to go about it. I'm more concerned about whether we should...

This issue has come up in a couple of conversations I've had recently, in different contexts. My main concern is about the concept of intent codes: if intent codes restrict how users can use otherwise completely identical data, I think they're a hindrance. It's not uncommon for images to be used for different purposes in different contexts. Some of our outputs can be visualised as RGB, but might not be intended for (or at least not only for) RGB display. I think this applies to different extents to all of these intent codes. My personal feeling here is that intent codes might provide a handy hint to users as to what the images represent, and might provide a hint to downstream applications as to how these images ought to be interpreted by default, but they should not prevent other interpretations where these are reasonable (i.e. dimensionality, data type, etc are all compatible with that other interpretation).

One recent conversation was about the -vector output of tensor2metric, which allows negative values in its output since these are 3D vectors, but this causes issues when these are interpreted as DEC maps (which by default, they can be) and displaying them in other packages than mrview. We could have an explicit option to tensor2metric that would produce a positive semi-definite RGB image, with the appropriate NIFTI_INTENT code set, but it couldn't then be used for e.g. tractography like the current output can. I would expect other packages to at least provide the option to display images that can legitimately be displayed as RGB as such, even in the absence of any intent code.

Things are different for tools like tckmap -dec or fod2dec, since these are already in the right RGB representation (at least, no negative values). It would be fair to set the NIFTI_INTENT_RGB_VECTOR code then. But I would still have a few concerns about doing this:

If I interpret the NIfTI standard correctly (not obvious, see below), we'd need to set dim[4]=1 and dim[5]=3. This isn't what we currently do: we use the first available non-spatial dimension for the RGB components, i.e. dim[4] (in NIfTI parlance; this is axis index 3 in MRtrix). If we were to be fully compliant with the standard, that would need a lot of modification to our handling. Should we modify all the relevant executables to always produce a 5D, X×Y×Z×1×3 image, irrespective of whether the output image is NIfTI? That would break the handling of existing images. Should we modify the NIfTI output only for images with that intent code? That would also require remapping axis 3 to axis 4 internally within the MRtrix3 codebase, which will inevitably lead to some weird and very unexpected outcomes. I'm not sure there's an easy fix here.
How do we handle these images if passed through generic commands like mrcalc or mrconvert? Some operations might preserve the RGB-ness of the data, others definitely won't. but it's not necessarily always trivial to figure this out upfront. How do we deal with this?

So to cut a long story short, I'm not sold on these intent codes...

Related queries here are:

Are intent codes widely used and adhered to in other packages and in general?
Are there existing tools that allow setting or modifying the NIfTI intent codes?

On a different note, something sounds a bit off in the comments for these fields. I'm assuming this is a mish-mash of the previous NIFTI_INTENT_NODE_INDEX field, and the subsequent RGB / RGBA fields. I expect dim[1] → dim[3] should be the XYZ dimensions, not 1...? But it does call into question what the expectations are exactly, if this bit is wrong.

  /*! To signify that the value at each location is a node index, from
     a complete surface dataset.                                       */

#define NIFTI_INTENT_NODE_INDEX   2002

/*! To signify that the vector value at each location is an RGB triplet,
     of whatever type.
       - dataset must have a 5th dimension
       - dim[0] = 5
       - dim[1] = number of nodes
       - dim[2] = dim[3] = dim[4] = 1
       - dim[5] = 3
    */

#define NIFTI_INTENT_RGB_VECTOR   2003

 /*! To signify that the vector value at each location is a 4 valued RGBA
     vector, of whatever type.
       - dataset must have a 5th dimension
       - dim[0] = 5
       - dim[1] = number of nodes
       - dim[2] = dim[3] = dim[4] = 1
       - dim[5] = 4
    */

My sense is that intent codes provide nice hints. I know that MRIcroGL and FSLeyes use them to provide a good initial display, and could be used by other tools. I think the more widely they are used, the better they can be supported.

Regarding your confusion regarding intent codes 2001..2005

The codes were originally created for GIfTI meshes. Here, they refer to the vertex index of a mesh. The vertex index of a mesh has only one dimension (the vertex number). Therefore, dim[2] = dim[3] = dim[4] = 1.
FSL has adopted these same codes for three dimensional NIfTI images. For these, dim[1], dim[2], dim[3] refer to the 3 spatial dimensions (columns, rows, slices) of a voxel image.

In the formal definition, NIfTI always reserves the first there dimensions for space, and the fourth for time. By this definition, DWI scans should actually be stored with dim[4] = 1 and dim[5] = number of volumes (though one could argue the different directionals are sampled sequentially in time). Likewise, RGB vectors should always have dim[4] = 1 and dim[5] = 3. However, in practice, the fourth dimension is simply used for number of volumes. Since FSL saves RGB with dim[4] = 3, I have tended to just look at number of volumes, regardless of whether they are specified as dim[4] or dim[5].

I think intent codes are only used by some tools. However, supporting them will help adoption and can help ease users lives.
- The NIfTI header is a fixed binary structure, so it is easy to work with.
- For Matlab, I like nii_tool.
- For a graphical user interface, you can use MRIcroGL's "Show Header" window (see below)- the Statistics tab has a pull-down menu. If you modify a header, you can use the File/Save menu item to save the revised header.
- For Python, you can use the nibabel get_intent() and set_intent():
```
import nibabel as nib
img = nib.load('sag_int_36sl_21.nii')
hdr = img.header
img.header.get_intent() 
```

intent

My sense is that intent codes provide nice hints.

That's essentially my understanding of what they should be about. However, the specific example you link to in your original post suggests that in some cases, these are used as more than mere hints. Presumably the only issue there was that the intent code was not set as suggested here? And this resulted in the user being unable to display the image as RGB, not even by clicking on the right settings? The problem as I see it is that unless you mandate that all software packages use these intent codes consistently, users are going to run into these issues routinely with no simple way to rectify the issue. You linked to some applications that allow users to 'fix' their intent codes, but I would argue these are not necessarily all that straightforward (although the MRIcroGL route looks pretty neat :wink:), and it's certainly not all that obvious to users not already steeped in NIfTI that they might even need to tweak their intent codes to be able to finally display or process their data correctly.

And as I hinted in my previous post, some of our commands may generate data where the appropriate intent code is obvious, but it's not obvious what to do about generic handling of these images. For example, one of the intent codes is NIFTI_INTENT_PVAL. What if we processed such an image with this command:

$ mrcalc pval.nii -log log_pval.nii
mrcalc: [100%] computing: log (pval.nii)

What should the intent code be? Assuming the original intent code was NIFTI_INTENT_PVAL, ideally the output intent code should be NIFTI_INTENT_LOGPVAL. But the -log operator could be used in all sorts of different contexts, and in combination with all sorts of other operations. What if we did this:

$ mrcalc 1 pval.nii -sub -log log_pval.nii
mrcalc: [100%] computing: log ((1 - pval.nii))

Should we expect mrcalc to figure out that whether or not this should still have the NIFTI_INTENT_LOGPVAL code? Or should it produce NIFTI_INTENT_NONE on the understanding that the operation involved more than a simple log(p)? I feel this places a massive burden on the developers to consider all (reasonable) use cases to avoid leaving users with badly formed intent codes that may cause problems downstream. In my opinion, these intent codes may be appealing for whatever process is consuming the final result (e.g. the display software) if they can assume it is well-formed, but this can only really work when every stage of the pipeline to get there has been carefully crafted with this in mind. And as this simple example illustrates, this is no trivial undertaking.

This might still be worthwhile if there was a clear advantage to doing this, beyond just providing a hint to users and/or software. Personally, I think filenames are a much clearer way to signal intent for users, and software should not make any assumptions about intent if the user has made their intention clear by providing the data as input for whatever purpose, as long as the data can reasonably be processed. But that's just my 2¢...

There are other issues with these that are more specific to MRtrix:

If we go down this route, we should also support things like NIFTI_INTENT_SYMMATRIX for DTI data (and probably other things too). But our storage order differs from NIfTI in that respect, and we use the 4th dimension to store the coefficients, rather than the 5th as mandated in NIfTI. And as with vectors, the coordinate system will likely differ as well (though the standard doesn't say much about that). So supporting that introduces large changes and incompatibilities with existing data.
MRtrix natively supports a much wider ranges of data types than NIfTI (DICOM, mif & MGH being the main ones), and providing support for intent codes across all these formats is going to be messy. And it's not just intent codes, but also storage conventions themselves that differ between formats (e.g. along which axis should coefficients be stored, in which order, which coordinate system, etc).

So with all this in mind, I'm reluctant to go down that road. What I might be comfortable with is adding an option to e.g. mrconvert to set the intent code, which would be ignored unless the output is actually in NIfTI format. That would allow users to explicitly override the default NIFTI_INTENT_NONE if they so wish, placing the onus on them to get it right for their own application. Not sure how the other devs feel about this, but it just strikes me as a lot of effort for very little practical gain, and potentially with a huge maintenance burden as more and more use cases come up where the intent codes don't come out as they should...

What I might be comfortable with is adding an option to e.g. mrconvert to set the intent code

Having now been reminded of the issues around intent codes & NIfTI dimensions, this is exactly where my mind went right as I got to this point. But as you describe, it would need to still be ignored on load.

It would also be of benefit to have a forum Wiki page on e.g. "Converting MRtrix3 outputs for other softwares", where there would be copy&paste examples of permuting axes & setting intent codes, could describe the -strides -1,+2,+3 trick, reordering tensor volumes, taking eigenvector absolute values for DEC display, and it would be open for users to expand with their own specific examples.

By the way, I'm not (really) trying to shut down the discussion, but I do have serious misgivings about intent codes, as you'll have gathered from the above. If there are really important use cases where NIfTI intent codes are crucial, I'd like to hear about them. But for now, I really think the path of least resistance for us is simply to allow users to set these intents codes on the command line.

For something like this, I'd advocate for an environment variable, similar to the approach taken for selecting DICOM studies (the DICOM_XXX variables), or setting the random seed (MRTRIX_RNG_SEED). This would allow usage such as:

NIFTI_INTENT=RGB_VECTOR tckmap in.tck -dec dectdi.nii

The advantage is that this allows users to modify the backend NIfTI handling for any MRtrix applications without cluttering up the list of standard command-line options.

And we could also set the default intent code in the application itself in those cases where we know it is appropriate, e.g. for tckmap -dec or dwi2dec, while still allowing the user to override that default. I reckon that might tick most boxes...?

It would also be of benefit to have a forum Wiki page on e.g. "Converting MRtrix3 outputs for other softwares", where there would be copy&paste examples of permuting axes & setting intent codes, could describe the -strides -1,+2,+3 trick, reordering tensor volumes, taking eigenvector absolute values for DEC display, and it would be open for users to expand with their own specific examples.

:+1:

For something like this, I'd advocate for an environment variable, similar to the approach taken for selecting DICOM studies (the DICOM_XXX variables), or setting the random seed (MRTRIX_RNG_SEED). This would allow usage such as:

NIFTI_INTENT=RGB_VECTOR tckmap in.tck -dec dectdi.nii

The advantage is that this allows users to modify the backend NIfTI handling for any MRtrix applications without cluttering up the list of standard command-line options.

The trouble there is that it might not be only setting of the intent code that the user needs to do for external compatibility: they may additionally need to shift RGB from the fourth to fifth axis, or e.g. take absolute values. A -nifti_intent option added to mrconvert only wouldn't be that much clutter, and the Wiki page would demonstrate how command outputs can be piped to mrconvert in order to do the requisite conversion (example usages could be added to the tckmap / tckconvert help pages also).

Sure, it would definitely require a Wiki page to document how to use that to produce outputs that are actually compliant. But I don't think that necessarily precludes allowing other commands than mrconvert to also set the intent code. In practice, people will just follow the guidance on the Wiki page anyway.

Where it might cause problems is if users decide to set that environment variable in a more permanent way (e.g. run export NIFTI_INTENT=XYZ in the terminal, or directly in their .bashrc). Using an explicit command-line option has a definite advantage there - though to be fair that limitation applies to most of our other environment variables too.

Also, if I understand @neurolabusc's previous comments, the NIFTI_INTENT_RGB_VECTOR wasn't originally intended for regular images anyway, and is already being used in non-standard ways by FSL (amongst others, presumably) - including storing the RGB coefficients along the 4th axis. So if we stick to the standard strictly, we shouldn't be storing RGB images with that intent in the first place. If we relax our interpretation and adopt the conventions that other packages are actually using in practice, we don't need to shift the coefficients to the 5th axis.

@jdtournier it does seem the FSL team has been using the Intent codes creatively. However, it does allow their tools and viewers (like FSLeyes) to give precedence to the canonical solution for most datasets. For MRIcroGL, I have simply been cloning their behavior.

I do think a rapid consensus could be agreed that would refine the description of the intent codes to handle how they are used in practice (as this fills a niche) and define a couple new ones to describe the different methods to describe how different tools natively describe their data. For example, there is no consensus regarding whether the DTI matrix form stores the upper or lower triangle. Why not at least provide intent codes that explicitly identify the convention used. This way, all tools could use their native format, but it would allow an easy mechanism to detect when a user assumes interoperability between tools.

MRtrix3 / mrtrix3

ENH NIfTI: Use NIFTI_INTENT_RGB_VECTOR when appropriate #2217