Transformation Specification

[constantinpape]

This is a feature request towards v0.2 of the multi-scale image spec: Support (affine) transformations for images to apply these on the fly (e.g. to map images to a shared coordinate system).

So far, I have come around two different implementations that contain a proposal for this:

Open Organelle

{
"datasets": [
      {
        "path": "s0",
        "transform": {
          "axes": ["z", "y", "x"],
          "scale": [1, 1, 1],
          "translate": [0, 0, 0],
          "units": ["nm", "nm", "nm"]
         }
       }
   ]
} 

The transformation metadata is also duplicated in the individual datasets.

I2K Tutorial

Same as the open organelle proposal, but the transformation is defined for on the dataset level, not for each individual scale:

{
"datasets": [...],
"transform": {...}
}

The transformation metadata is not duplicated.

(I have left out the nested multiscale etc. in both cases to make this more readable.)

I would be happy to develop this into a PR to the spec, but it would be good to have some feedback first:

• Is there a competing approach we should take into account? Any general critique?
• Should the transformation be defined per dataset or per scale level (I think per scale level actually makes more sense).
• Do we add rotation and shear to transformation, so that a full affine trafo can be specified?

@joshmoore @frauzufall @kephale @axtimwalde @d-v-b @tischi @jni (please tag anyone else who might be interested in this)

---

Original text from #12:

Originally discussed as part of the multiscales specification, scale metadata related to each volume in a multiscale image needs storing for the proper interpretation of the data.

see also https://github.com/ome/omero-iviewer/pull/359 in that each level should be strictly smaller than the last.

[jni]

Note that there is a scale "discussion" ;) in #12.

Some considerations:

• "axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?
• In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?
• From my perspective, and with utmost respect to affine transforms 😂, I think scale/translate/physical units is priority 1, and affine should come later. Having affine and scale/translate in the same discussion complicates it unnecessarily (do you represent affine as a matrix or as the decomposed values? How do you describe rotations in 3D? etc)
• With regards to axis order etc, I think (?) the original OME model uses things like PhysicalSizeX and PhysicalSizeY, which I hope we can avoid in ngff but I dunno, it sounds like compatibility to existing OME metadata is a concern @joshmoore?

[constantinpape]

Note that there is a scale "discussion" ;) in #12.

Ok, good point I did not see that one and it's indeed related.

* "axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?

Yes, axes is the dimension order as it refers to the values for this transformation. One could also store this at a higher level in the metadata and then assume that the transform values always refer to this axis order.

• In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?

• From my perspective, and with utmost respect to affine transforms joy, I think scale/translate/physical units is priority 1, and affine should come later. Having affine and scale/translate in the same discussion complicates it unnecessarily (do you represent affine as a matrix or as the decomposed values? How do you describe rotations in 3D? etc)

So the idea here would be to describe one transformation per image. I think chaining transformations is currently out of scope. Regarding affine vs. translation / scale: Sorry, I should have expanded this more, but I thought about the following design: Transform has the following 4 fields:

"scale": [],
"translate": [],
"rotate": [],
"shear": []

They are all optional, where their absence indicates a scale of 1, no translation, no rotation etc. This way the full affine transformation can be specified but one can also just specify the scale, translation etc. Then indeed one would need to agree on how to describe the rotation and shear (euler angles for the rotation?). With the MoBIE hat (and I think also BDV hat) on, it would be pretty important to be able to describe the complete affine right away.

[d-v-b]

"axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?

I decided to put "axes" in there because I work alongside people who use languages / libraries that use different array indexing schemes, and so we ran into lots of annoyances due to ambiguity about axis names (is the first axis z, or is it x?). axes also allows you to convert the transform object into a key-value pair with meaningful keys (in fact, I considered a key-value pair with semantic keys for the transform object.). And I think this information does belong with scale and translate, since it clarifies their meaning.

And I think every scale level should have the transform listed. The principle for that is "multiscale is just a collection of single-scale datasets => multiscale metadata is just a collection of single scale metadata".

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

[constantinpape]

And I think every scale level should have the transform listed. The principle for that is "multiscale is just a collection of single-scale datasets => multiscale metadata is just a collection of single scale metadata".

We modeled the metadata here after the open organelle layout, but at the time didn't see a need for a transformation for each scale level. But I have thought about this more since then, and I agree that it's better to have it for each scale level, which allows to display them separately.

[d-v-b]

I'm not 100% sure there is an absolute right answer here. One drawback of listing transforms in multiscales is that metadata gets duplicated -- the transform is specified once in dataset metadata, and again in multiscales, which is kind of ugly and could be a burden if transform metadata gets too big, or multiscales spatial metadata somehow differs from the dataset-level spatial metadata.

However, from an implementation standpoint, it's very convenient to access metadata from a single consolidated file (i.e., a single GET), rather than accessing it from multiple files (i.e., multiple GETs). This convenience was also a motivation for putting the the spatial transform stuff in multiscales, and the access pattern advantages outweighed worries about duplicating information. But maybe other people reach the opposite judgment.

[joshmoore]

Note that there is a scale "discussion" ;) in #12.

Ok, good point I did not see that one and it's indeed related.

And has basically nothing in it. More than happy to let @constantinpape take the lead. Closing #12.

[constantinpape]

One drawback of listing transforms in multiscales is that metadata gets duplicated -- the transform is specified once in dataset metadata

We didn't put it in the dataset metadata at all and only in multiscales. I think it's definitely benefitial to have it consolidated in multiscales to reduce the number of request and make it easier to access.

I don't feel strong about having it with the individual datasets or not; I think for what we design the viewer would always access the multiscales. This would be something for @joshmoore to weight in on how he feels about duplication of metadata in the spec.

[d-v-b]

I don't feel strong about having it with the individual datasets or not; I think for what we design the viewer would always access the multiscales. This would be something for @joshmoore to weight in how he feels about duplication of metadata in the spec.

I think it's critical that individual datasets always have spatial metadata. The multiscales metadata cannot replace the individual spatial metadata of a single dataset, because datasets should be portable -- I should be able to open a single dataset, outside of the context of a multiscale pyramid, and get valid spatial information.

[jni]

I should be able to open a single dataset, outside of the context of a multiscale pyramid, and get valid spatial information.

I agree very much with this design constraint. To me it certainly feels more important than the problem of metadata duplication, which can be managed.

[constantinpape]

Ok, then I think we have clear perference: transformation metadata must be specified in the individual datasets and in multiscales. Now, we only need to define the actual trafo metadata :).

[axtimwalde]

What do you guys think of speccing out a flexible extensible trafo meta data format with clear exit points when something is not supported? Our very old approach to this is the TrakEM2/ Render format (http://www.ini.uzh.ch/~acardona/api/mpicbg/trakem2/transform/CoordinateTransform.html and https://github.com/saalfeldlab/render/blob/master/docs/src/site/markdown/data-model.md). The gist

transformation {type:String, data:String}

and

list{[transformation|list]}

The entire spec here is very Java centric (type is the legit class name), and data encode all sorts of stuff into a String which is limiting in how much you can map into it. Also, a reference to a file/ dataset (for deformation fields or coefficients) is not intended. My heart would be a lot lighter if we could focus on generating such a spec including all transformations that we know of or have implementations of at this time instead of drooling over how to write parts of affine transformations.

[constantinpape]

@axtimwalde If I understand your proposal correctly, some example transformations could look ike this:

{
  "transformation": {"type": "affineBdv", "data": "..."},  # affine transformation in bigdataviewer format
  "transformation": {"type": "eulerElastix", "data": "..."}  # euler transformation in elastix format
  ...  
}

and we would somewhere list the available types and the corresponding format for "data".

And list would be a list of these transformations that allows chaining them?

[axtimwalde]

Approximately, but the data : String approach is very limiting and it should therefore be flexible what to put into the parameters. Often, that would be something like

"transformation" : {"type" : "scale", "parameters" : [4, 4, 40]}

[constantinpape]

@axtimwalde I think this is a good solution, because it also makes it easier to extend the transformations later. (E.g. to support #30). And it would allow to very simply specify scale and translation and concatenate other transformations, @jni.

I am also not so sure whether adding the axes and units fields (or similar) to the transformation is such a good idea. This metadata refers to the data and not specifically to the transformation. How would we deal with inconsistent values between the metadata for the transformation and the data?

Also, in the current form, the image spec always assumes 5D data. However, the transformations are only valid for the 2 or 3 spatial dimensions. This is something we need to keep in mind too, but I think we first need to discuss the 5D requirement more, see #35.

[d-v-b]

My heart would be a lot lighter if we could focus on generating such a spec including all transformations that we know of or have implementations of at this time instead of drooling over how to write parts of affine transformations.

I disagree :) I don't think trying to come up with a spec that covers all known / possible nonlinear transformations is feasible or a good use of time. In my view, as soon as you go down the road of applying a nonlinear coordinate transformation to your data, you are working with specialized tools and the developers of those tools should be allowed to use whatever metadata they need. This only requires that the ngff transformation spec be permissive.

On the other hand, I think it does make sense to be formal about linear coordinate transformations, because they are universal -- every image processing library understands affine transforms (and most visualization tools can display data on an affine-transformed grid, with varying levels of support for shear and rotation). Because linear transforms are universal, I think it does make sense to come up with (or copy) a readable, compact description of linear transforms. This could either be a list of named transformations, like scale and translate, or a full matrix. We just need to ensure that we include information that gives meaning to the rows and columns of this matrix.

I am also not so sure whether adding the axes and units fields (or similar) to the transformation is such a good idea. This metadata refers to the data and not specifically to the transformation. How would we deal with inconsistent values between the metadata for the transformation and the data?

I don't think axes and units describe the data; instead, axes and units describe the grid on which the data is sampled. The spatial transform under discussion also describes this grid. In particular for shears and rotations it's very important to explicitly name the dimensions of the input space and the dimensions of the output space -- in medical applications, images might be acquired in machine coordinates and transformed into anatomical coordinates.

[axtimwalde]

Not every image/ array processing library supports affine transformations, translations or rigid transformations have fewer parameters than affine transformations which justifies treating them differently than affine transformations, but default decomposing affine transformations makes little sense in practical applications that use affine transformations and is incredibly verbose. Axes in data space are usually a mixture of several physical dimensions, i.e. a scanner scans a spatial dimension while spending time, so the 'correct' axes label would be "x + t" with a unit of "0.4um + 0.0000s" or something. It is quickly clear that particularly the qualifiers in the unit field are clearly parts of a transformation. I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions (if you want). A transformation can transform between all sorts of spaces, not only physical and data. I feel bad about muddling a "default" axes permutation (another subset of affine transformations) into the spec by naming the axes of the transformation, so they can be used for data with axes that have the same name. That is a very complicated way of specifying the axes permutation into a 'default' data space in which some other transformation can be applied. I think it would be great for transformations to have names, e.g. "xycz2xyzc" for one such axes permutation. A 'standard library' of transformations may only contain affine transformations and subsets thereof at first and may be extended by other classes of transformations and reference implementations as needed.

[constantinpape]

Thanks for the detailed comments @d-v-b @axtimwalde. I will try to go through it point by point:

Transformation Specification:

I don't think trying to come up with a spec that covers all known / possible nonlinear transformations is feasible or a good use of time. [...] developers of those tools should be allowed to use whatever metadata they need.

Not every image/ array processing library supports affine transformations, translations or rigid transformations have fewer parameters than affine transformations which justifies treating them differently than affine transformations, but default decomposing affine transformations makes little sense in practical applications that use affine transformations and is incredibly verbose.

I think we are on a very similar page here. The transformation spec

{"type": "some_transformation_type", "parameters": {...}}

allows to specify custom transformations. But to support interoperability, we would provide a list of "standard" transformation types (e.g. translation, scale, rigid, affine) with a strict definition. Developers can still use their custom transformations, but then there is no guarantee other tools can parse it. (At some point the list of standard types could be extended with common custom transformations, but let's not get ahead of ourselves here).

Axes and units:

I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions

This makes a lot of sense. The only open question for me is what to do about image datasets that don't specify a transformation, but for which one still needs to specify the axes. Potential solutions:

1. All image datasets must specify a transformation
2. Add additional metadata to specify the physical dimensions in case no transformation is given

I would prefer 1, but this would be a breaking change with the current spec.

Axes permutations:

I feel bad about muddling a "default" axes permutation (another subset of affine transformations) into the spec by naming the axes of the transformation, so they can be used for data with axes that have the same name. That is a very complicated way of specifying the axes permutation into a 'default' data space in which some other transformation can be applied.

I agree with this.

My point here was that it's unclear how to indicate which axes of the input data the transformation is applied to. Note that in the current spec data is always 5 dimensional, which makes this necessary. If we drop the 5D requirement from the spec, see also #35, we could demand that transformation and data have the same number of dimensions and avoid this issue.

[d-v-b]

I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions

If the axes of the source space are not named, then information is needlessly lost. Imagine point-scanning an image of a region in space, then scanning the same region after rotating the sample 90 degrees (i.e., transposing the scan axes). Without semantics for the source axes (e.g., names like scan_y, scan_x), it is cumbersome to distinguish these two images. But I agree that a default unit of px makes sense for the source space.

Potential solutions:

1. All image datasets must specify a transformation
2. Add additional metadata to specify the physical dimensions in case no transformation is given

I would prefer 1, but this would be a breaking change with the current spec.

I also prefer 1, and I note that option 2 is just shoehorning in a specification of a transform ("physical dimensions" is just a scale transform), so these two options aren't really different.

[tischi]

+1 for "all data sets must specify a transformation" (aka a mapping from data space to physical space)

As @constantinpape pointed out, to practically proceed further I guess the question of the dimensionality of the data space indeed seems of quite high priority.

[axtimwalde]

Thanks for picking this up @d-v-b and @constantinpape.

If the axes of the source space are not named, then information is needlessly lost. Imagine point-scanning an image of a region in space, then scanning the same region after rotating the sample 90 degrees (i.e., transposing the scan axes). Without semantics for the source axes (e.g., names like scan_y, scan_x), it is cumbersome to distinguish these two images.

This is a transformation. Rotating the sample will most likely not be a perfect 90deg rotation either, so being able to do this with a flexible transformation spec instead of renaming axes would be much desired.

1. All image datasets must specify a transformation

This is compatible with specifying no transformation: No transformation means identity transformation. Named axes and units are specified for the target space of the transformation. No named axes means data space with unit "px". This is compatible with named axes and units for images because that means that the target space is achieved by applying the identity transformation after which "px" are "cm". The current standard case of scaled named axes to express the resolution is also compatible with this, the scaling as the transformation, the axes names and units are the target space.

[tischi]

I am wondering how a transformation from, e.g., a 3D data space that should be mapped into physical space with the dimensions XYC would look like? Did you consider this or are we currently assuming that all dimensions of the transformation are always spatial? I think an argument for supporting a data space XYC was that some people have data with lots of channels and would like to configure the chunking such that all channels for one location XY are loaded as one chunk.

[tischi]

Let's say I want 5 micrometer voxel spacing in the physical space and translate 10 micrometer and have XYC data space. Would something like this be the idea?

affine = 
{ 5, 0, 0, 10,
  0, 5, 0, 10,
  0, 0, 1,  0 }
axisLabels: {"x", "y", "c"}
axisUnits: {"micrometer", "micrometer", "channel"}

[joshmoore]

https://github.com/ome/ngff/issues/28#issuecomment-784168157 I think scale/translate/physical units is priority 1

:+1: And that may be part of "drawing a line" through this issue as well.

https://github.com/ome/ngff/issues/28#issuecomment-784168157 but I dunno, it sounds like compatibility to existing OME metadata is a concern @joshmoore?

Eventually there will be need to be something to cover all the OME metadata. If it's handled in one of the preliminary more binary-focused specs, then :+1: and those fields need not appear verbatim. If not, eventually there will need to be an OME-prefixed block to take the rest.

[constantinpape]

@joshmoore I think we can distill a pretty good proposal out of the current discussion that also covers some more transformations and would be extensible to more custom ones. I'd be happy to formalize it, but it hinges on a decision about the dimensionality in #35.

#28 (comment) I think scale/translate/physical units is priority 1

+1 And that may be part of "drawing a line" through this issue as well.

[constantinpape]

Let's say I want 5 micrometer voxel spacing in the physical space and translate 10 micrometer and have XYC data space. Would something like this be the idea?

affine = 
{ 5, 0, 0, 10,
  0, 5, 0, 10,
  0, 0, 1,  0 }
axisLabels: {"x", "y", "c"}
axisUnits: {"micrometer", "micrometer", "channel"}

@tischi I agree that we should think about how to deal with non-spatial axes in the transformation spec. But I see some issues with your specific proposal:

• I don't really see any use-case for applying an actual affine transform to the non-spatial axes. So this should always be the identity transform, but when it's meshed up with the spatial axis this opens the room for errors.
• This gets tough when we have 3 spatial dimensions, because then we need to define a 4d (or even 5d) affine.

I am not quite sure what's a better approach yet, my plan here was to wait for a decision about the data dimensionality, then distill what we discussed for the transformation spec so far and think about how to fit non-spatial axes into that. (I am happy for any more suggestions on how to tackle this problem in the meantime of course.)

[axtimwalde]

@tischi

affine = { 5, 0, 0, 10, 0, 5, 0, 10, 0, 0, 1, 0 } axisLabels: {"x", "y", "c"} axisUnits: {"micrometer", "micrometer", "channel"}

my favorite would be something like

"transformation" : {
  "type" : "sequence",
  "name" : "scale and shift but leave the channels alone",
  "data" : [{
    "type" : "scale",
    "data" : [5, 5, 1]
  }, {
    "type" : "translation",
    "data" : [10, 10, 0]
  }]
},

"axis" : [
  {
    "label" : "x",
    "type" : "space",
    "unit" : "nm"
  }, {
    "label" : "y",
    "type" : "space",
    "unit" : "nm"
  }, {
    "label" : "c",
    "type" : "custom name for what is meant with channels as long as it isn't wavelength which should be space"
    }]

[axtimwalde]

or

"transformation" : {
  "type" : "scaleShift",
  "name" : "scale and shift but leave the channels alone",
  "data" : [[5, 5, 1], [10, 10, 0]]
},

"axis" : [
  {
    "label" : "x",
    "type" : "space",
    "unit" : "nm"
  }, {
    "label" : "y",
    "type" : "space",
    "unit" : "nm"
  }, {
    "label" : "c",
    "type" : "custom name for what is meant with channels as long as it isn't wavelength which should be space"
    }]

[tischi]

@axtimwalde And would the "label" need to be chosen from a defined set: "x", "y", "z", "c", "t" or could that be free text? I'd naively think it should be from a defined set, because otherwise I do not see how the viewer would know which axis should be displayed, e.g. horizontally ("x") on the screen.

[d-v-b]

I do not see how the viewer would know which axis should be displayed, e.g. horizontally ("x") on the screen.

I want to push back on this line of thought. Axis names are just names. They do not mean anything intrinsically. We should not expect that everyone wants to call their axes "x" "y" and "z", and we should not expect that an axis called "x" should be intrinsically mapped to the horizontal axis of a screen.

[jni]

I agree 110% with @d-v-b. Viewers can decide which world axis to map to a screen axis — and in fact the mapping is in no way direct — ie when rotating a volume in 3D the axes point every which way on the screen. Or you might want to view a kymograph, in which case t is mapped to the horizontal axis on screen.

[axtimwalde]

Great, we all agree, but I think we are at different levels of internalization without realizing this.

It is useful for the viewer to understand what the type and the units of an axis are. So I think labels can be arbitrary, but the type should be provided, indicating that it is either one of the seven base units or some custom thing like "burger count" or "brain compartment". The transformation defines how to get from default data space into this labeled space.

We should also settle on the handedness of our spaces, so viewers can display 3D objects without accidental flipping. We typically assume right handed in image processing when reading axes in natural order (x, y, z) or left-handed when reading in Python order (z, y, x). This has to be spelled out. If the data comes in wrong handed, e.g. from a microscope that scans left to right but from bottom upwards, then the transformation must settle this (e.g. {"type" : "flip", "data" : 1}).

Defaults with no transformation (meaning identity) and no axes labels (meaning some standard, e.g. x,y,z,t,c) and no axis types (meaning "null" or "data") and no axes units (meaning "px") are incredibly useful.

[tischi]

we should not expect that an axis called "x" should be intrinsically mapped to the horizontal axis of a screen.

@d-v-b yes, that's right, I am taking this back, this was a bad example and wrongly formulated. I think the way @axtimwalde puts it in terms of the "handedness of our spaces" is more what I was getting at.

[tischi]

Defaults with no transformation (meaning identity) and no axes labels (meaning some standard, e.g. x,y,z,t,c)

I think this is an important aspect, because to my best knowledge, a very large fraction of the current bioimage analysis software ecosystem relies on an image data model with the conventional "physical" interpretation of the axis labels x,y,z,t,c. And I think it would be important for a fast acceptance of the file format that the current ecosystem can consume the image data presented in an ome.zarr.

[glyg]

As advised by @tischi in #35 here is a (corner) use case: Colleagues do 3D polarization microscopy, where they measure the orientation of a fluorescent birefringent marker. The raw data is thus 7D: time, 3 spatial coordinates, and 3 polarization angles (I think they quantify the angles as:

• rho : mean angle w/r to the acquisition plane x axis
• theta : mean angle outside of the acquisition plane
• psi : precession angle around the mean pola

{
    "axes": ["x", "y", "z", "rho", "theta", "psi", "t"],
    "units": ["micrometer", "micrometer", "micrometer", "radians", "radians", "radians", "minutes"]
}

For the transformations, this implies being able to specify a 6 axis transform, as the first 2 angles must rotate with the image.

[glyg]

While I'm in that corner, another, maybe less exotic case is single molecule localization microscopy, where we might want to keep a local measure of the localization error, so

{
    "axes": ["x", "y", "z", "sigma_x", "sigma_y", "sigma_z", "t"],
    "units": ["micrometer", "micrometer", "micrometer", "micrometer", "micrometer", "micrometer", "minutes"]
}

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/call-for-comments-on-brain-imaging-data-structure/50701/2

[thewtex]

In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?

👍 Since anisotropic pixel spacing and an offset is common with microscopy and other types of scientific images, supporting spacing and offset as standard spatial metadata associated with the image makes sense.

These two are built into an xarray.Dataset representation as discussed in #48.

A few other items for consideration:

Parameter precision turns out to be very important in practice with spatial transformations (lessons learned over the years in itk). Serialization and deserialization in decimal ascii in general will result in a loss of precision due to the differences in binary / decimal encoding and how IEEE float's are represented. This is most significant for transformations that involve rotations. In general, special effort has to be taken to avoid loss of precision, e.g. google double-conversion. So, it is best to store the transformation parameters in float64 arrays.

It works well to store a spatial transformation, identified by the transformation type and its parameters, along with an identifier of the coordinate system it transforms from and the coordinate system it transforms to. Then, the image holds in its metadata the identifier for its coordinate system. A few examples of this include DICOM spatial transformations 1 2 and WIP in NeurodataWithoutBorders (NWB).

Even for "simple" rigid transformations, there are a dozen ways that they can be interpreted unless all the information is there to interpret them. As @axtimwalde and I learned when examining the bioimage electron microscopy community prior to standardization, a dozen researchers will somehow manage to all pick the different ways 🥚 🍳 🍞 🥖 🥪

In terms of possible transformations, ITK has defined a nice phylogeny (attached). SpatialTransformations.pdf

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/fiji-hackathon-2021-big-data-days-a/53926/7

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-early-september-2021/55333/14

[bogovicj]

Here is an early but concrete proposal @axtimwalde and I wrote based on the discussion above that reflects our preferences (of course).

https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF

[constantinpape]

Here is an early but concrete proposal @axtimwalde and I wrote based on the discussion above that reflects our preferences (of course).

https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF

Thanks, John! Overall I really like this proposal and its extensible and already covers some of the non-linear use-cases of #30.

A couple of comments:

• The Axis types section has diverged a bit from the current solution in v0.3, see https://github.com/ome/ngff/issues/35#issuecomment-910659321. But I think it's feasible to combine the current version and the proposal.
• How would you specify which axes the transformations act on, e.g. if we have axes tczyx and an affine? How would the software now that its applied to zyx? Related to this, it might not be a good idea to hard-code xyz axis order. Maybe adding a field that specifies which axes the trafo acts upon would solve this?
• For d_field and p_field we could also allow (or demand) storing the parameters as a zarr array in the same container, which would help in keeping all relevant data together.

In addition, following the discussions in #53, there seems to be strong support to specify a transformation per scale dataset. I think the easiest way to do this would be

"datasets": [
  {"path": "s0", "transform": {"type": "affine", "affine": [...]}},
  {"path": "s1", "transform": {"type": "affine", "affine": [...]}},
  ...
]

[tischi]

@bogovicj @axtimwalde

Thanks a lot for fleshing this out! ❤️

I also have a couple of questions:

Transformations are specified as forward transformations, i.e. a transformation transforms a vector from source into target space

What's the motivation to specify in this direction? For example I think in the elastix registration software they specify from target to source. I am not saying that I would prefer to specify it in the other direction, I am just curious!

Axes types

The axes right now have a field called label. In the case of a spatial axis, you gave an example where you gave the axis the label y. I wonder, is this (a) just some text, such that we also could have called it foo, or (b) the label contains information about how this axis should be "arranged" in the right-handed spatial coordinate system? Personally, I'd be OK with assigning the "labels" x,y,z, and t specific meanings, because I would say this is a very well established convention in many physics text-books and we live in a physical world. But I think, e.g., @jni had concerns about this?

[bogovicj]

Thanks for the feedback Constantin! @constantinpape

On axes:

I think it's feasible to combine the current version and the proposal.

I agree. The only "big" difference that I see to what you describe in #35 is the "type" (spatial, time, etc.). We might also see the nrrd spec. and as you say there:

it would be easy to have an equivalent solution using 3 lists, e.g. axes, axes_label and unit.

which I would be very happy with.

[pd]-fields

For d_field and p_field we could also allow (or demand) storing the parameters as a zarr array in the same container, which would help in keeping all relevant data together.

I think this is a great idea.

"which axis does my transform operate on"

How would you specify which axes the transformations act on

You've pinpointed a hard thing. I'm not sure, and am open to ideas.
Here's a wild collection of ideas ranging from

"shove all the transformations together"

{
      "transform": {
        "type": "affine",
        "affine": [
            2.0, 0.0, 0.0, 10.0,
            0.0, 0.5, 0.0, -5.0,
            0.0, 0.0, 1.0, 0.0
        ]
      },
      "axisTypes": ["spatial","spatial","channel"],
      "axisLabels": ["X","Y","C"]
  }

to "split by type"

  "axes": [
      {
          "type":"spatial",
          "labels":["X","Y"],
          "indicies":[0,1],
          "transform": {
              "type":"affine",
              "affine":[2.0, 0.0, 10.0,
                        0.0, 0.5, -5.0 ]
          },
          "unit":"nm"
      },
      {
          "type":"channel",
          "labels":["C"],
          "indicies":[2],
          "transform": {
              "type":"identity"
          }
      }
  ]

the former is simpler, the latter "clearer" somehow and does not let one "mix" different types of axes. I'm even tempted to allow a couple of options ranging from simple to general.

[bogovicj]

Thanks @tischi !

transform inverses

What's the motivation to specify in this direction? For example I think in the elastix registration software they specify from target to source.

Exactly! Our thinking was that it fits most people's (? certainly my) mental model better. In the case of elastix output, we have in mind to do something like:

{
   "type" : "inverse_of",
   "transformation" : {
      "type" : "bspline_dfield",
      "data" : "the parameters..."
   }
}

axes

(a) just some text, such that we also could have called it foo, or (b) the label contains information about how this axis should be "arranged" in the right-handed spatial coordinate system?

Good point. I was thinking (a), kind of. More we mean it to be a way for people / software that generates data to encode something about it. I'm thinking about what you say here. Sometimes the first index of the volume increasing is arbitrary (call it x or foo). Sometimes is not, call it anterior-posterior, A-P, or left-right / LR or whatever.

So, I wasn't thinking about (b), but maybe we should.

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

me too.

[tischi]

So, I wasn't thinking about (b), but maybe we should.

Yes, I guess we need something to tell how the spatial axes are mapped onto the right-handedness, isn't it?

[tischi]

...or is that maybe implicit in the transformation, i.e. the right-handedness is a feature of sample space and not voxel space. Thus, maybe just the order of the spatial axes after the transformation defines the axes order and thereby the handedness (to this end, within the transformation one could "switch" axes if needed).

[d-v-b]

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

What does this mean exactly? I think wherever possible the semantics of axis names should not play any role in this spec. The spec should only "know" that axes have string names. I think mapping string names on to real life stuff is the job of tools that consume data, not the storage layer.

[bogovicj]

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

I understand this to mean: x,y,z are spatial dimensions and x cross y = z t is a time dimension

I think mapping string names on to real life stuff is the job of tools that consume data, not the storage layer.

How can the consuming software know that some spatial dimension is left-right, vs right-left, vs inferior-superior if the storage layer doesn't tell it?

[d-v-b]

I understand this to mean: x,y,z are spatial dimensions and x cross y = z t is a time dimension

If this kind of information is part of the spec, it creates interaction between axis names and axis types -- would it be a violation of the spec to have an axis called z that has type time? I think this kind of validation should be done by consuming software, not the storage layer.

How can the consuming software know that some spatial dimension is left-right, vs right-left, vs inferior-superior if the storage layer doesn't tell it?

If I understand it correctly, this kind of information pertains to the relationship between axes, not the mapping from axis names onto physical stuff, and so this seems in-scope for the spec.