Spatial registration of data

[campagnola]

I see good effort here to allow data to be temporally aligned, but there are many types of experiment where spatial alignment is crucial, and (as far as I can tell) this does not appear to be well supported by the spec or API.

Example use cases:

• Images acquired in a tiled mosaic could be automatically realigned for later display
• Likewise, images acquired at different magnifications or with different devices could be correctly scaled and aligned
• Electrode or photostimulation positions could be mapped to image coordinates
• ROI/cell coordinates and other anatomical markers could be stored in a standardized way
• The global coordinate frame for an experiment could be mapped to an atlas such as CCF or to stereotaxic coordinates

Suggested implementation details:

• In the same way we have a common timebase, there should be a notion of a common global coordinate system. This coordinate system origin might be defined arbitrarily for each experiment, and would simply allow data within an experiment to be related spatially.
• Each piece of data optionally contains enough information to map its own local coordinate frame (such as the rows/columns of an image) back to the global coordinate frame. For images this would usually be adequately described by a 3D affine transformation matrix, and for the position of an electrode tip or photostimulation spot, an xyz point might suffice.
• The API should provide functions that map between coordinate systems to allow the user to ask questions like:
  • how large is a pixel in this image?
  • what is the position/orientation offset between two images?
  • what is the row/column position within an image that corresponds to the location of an electrode tip?
• Transforms can be static or time-series.
• Coordinate transforms should be extensible, such that brave souls can implement their own transformations both in the spec and API. For example:
  • Some optics such as grin lenses may have distortions that cannot be corrected by a simple affine transform
  • Light sheet microscopy data may be acquired at oblique angles and with nonuniform pixel size
  • Resonant scanning images may likewise have nonuniform pixel size
• Making the coordinate mapping API extensible requires the availability of mapping functions that are agnostic to the form of the underlying transformations; the user should usually just ask for coordinates to be mapped from one frame to another, rather than asking for access to a transformation matrix.
• Some sensible suggested constraints for the global coordinate system:
  • The scale of the global coordinate system is in meters (eg, the position [1e-3, 0, 0] is one mm away from the origin along the x axis).
  • The axes are perpendicular.
  • Axis directions follow some common conventions wherever possible (for example, the z axis is usually parallel to the optical axis of the imaging system).

[campagnola]

Ok, the hackathon went well and we had many productive discussions. Here I will describe the plan that we came up with and that was presented to the group at the end of the hackathon:

Use cases and suggestions

• Image mosaic / experiment reconstruction - In an experiment involving multiple images taken at different locations and with different magnifications, electrodes placed at specific locations and moving over time, and/or laser photostimulation at specific locations, we must be able to load all data after the experiment and completely reconstruct the spatial configuration of the experiment at any point in time.
• Transformation API - should be easy from the Python API to map spatial coordinates between coordinate systems. For example, map the position of an electrode to a pixel in an image, or map the position of an image pixel to a standardized atlas. See ITK's transform classes for inspiration / cross-testing.
• Mouse head tracking - should be able to store raw data about the position of the mouse relative to its cage without attempting to relate it back to the position of the electrode relative to the brain. That is, the schema must be able to describe multiple unrelated coordinate systems.
• Eye tracking to retinotopic space mapping - should be possible to take pupil tracking data and map this to gaze direction -> visual space -> visual stimulus. This would allow to ask at any particular time, where does a particular pixel in a visual stimulus lie in the retinotopic space of the tracked eye.
• Mapping calcium imaging data to ISI/CCF - Calcium imaging experiments contain many different imaging modalities overlaid on the same region of cortex: intrinsic signal imaging, vasculature imaging, 2p structural imaging, 2p calcium imaging, and post-hoc tissuecyte imaging. Each of these carries an implicit coordinate system, and mapping between them is a common use case. We must be able to describe these coordinate systems unambiguously and the python API should reduce the burden of mapping between them. May involve the use of custom-specified nonlinear mappings.

[campagnola]

Schema changes

1. Addition of a CoordinateSystem class that allows declaration of coordinate systems that are used within the NWB:

   CoordinateSystem
       name: string
       description: string
       dimensions: integer number of dimensions, usually 2 or 3.
       units: list of strings describing the unit of length along each axis of the
           coordinate system
       transforms:  list of transforms that link this coordinate system to others defined 
           either in the same file, or to external, standardized coordinate systems such
           as Allen Institute's CCF. 

2. Addition of a /general/coordinate_systems group that contains instances of CoordinateSystem. Most NWB files would contain a small number (perhaps 0-2) of CoordinateSystem instances; a very common use case might be to have a single CoordinateSystem named "global" to which all other data in the file can be related. In some cases, however, it would be useful to declare multiple coordinate systems that are either linked by transforms (meaning we can map spatial coordinates from one CS to the other), or that are independent (we cannot map coordinates between these).

3. Addition of a TransformBase class and a collection of subclasses implementing commonly-used transforms:

   TransformBase
       source_cs
       destination_cs
   TranslateTransform
       offset
       source_cs
       destination_cs
   AffineTransform
       matrix
       offset
       source_cs
       destination_cs    

   Each transform specifies the coordinate system that it maps from (source_cs) and to (destination_cs), and also carries any parameters needed to mathematically specify the transformation. destination_cs could be either one of the coordinate systems defined in /general/coordinate_systems, or a reference to an external / standardized coordinate system.

4. Anywhere we store spatial data (such as in the electrode positions in an electrode table, or in a SpatialSeries or in the pixel locations in an image), it must be possible to unambiguously define the meaning of these coordinates. Each set of coordinates implicitly defines a coordinate system, so we seek to make this definition more explicit. This is done primarily by the addition of a transform subgroup wherever spatial data is found:

   • Image.transform and ImageSeries.transform would each contain a single instance of a transform with a destination_cs that references one of the entries defined in /general/coordinate_systems. The source_cs parameter would implicitly refer to the pixel/voxel coordinate system of the parent image. That is, the transform maps from pixel/voxel index (row, column) or (frame, row, column) back to the destination_cs.
   • Electrode positions are stored as (x,y,z) vectors in an electrode table. To relate these coordinates to the rest of the experiment, an Electrode.transform would map from the coordinates used in the electrode table to one of the entries in /general_coordinate_systems. For clarity, a set of standard conventions should be adopted that determine the meaning of the implicit source_cs used by Electrode.transform -- for example, a single-shank electrode might define this coordinate system as having its origin at the tip of the shank, and its +x axis pointing along the shank, in the direction of the tip. Similar conventions should be adopted for other types of electrode.
   • SpatialSeries.transform would be used as a more general approach, and it would be up to the experimenter to describe the meaning of the source_cs (although this would not always be necessary; sometimes just having the transformation is sufficient).

Note: In the current schema, it is possible to overspecify relationships between coordinate systems (for example, there may be multiple ways to map from one CS to another, and thus the mapping could be ambiguous depending on the path chosen. This could be mitigated in the Python API by detecting / warning about such conditions, or by refusing to allow them.

[campagnola]

PyNWB API examples

1. Coordinate systems defined in /general/coordinate_systems can be accessed from the NWB file:

   global_cs = nwb_file.coordinate_systems['global']

2. Image transforms can be set:

   image = ImageSeries(...)
   image.transform = AffineTransform(matrix, offset, destination_cs=global_cs)

3. Coordinates can be mapped through transforms:

   global_pts = image.transform.map(pixel_indices)
   pixel_indices = image.transform.imap(global_pts)

4. Transforms can be automatically composed together in order to map between any two coordinate systems, as long as they are linked in some way:

   image1.transform = AffineTransform(matrix1, offset1, destination_cs=global_cs)
   image2.transform = AffineTransform(matrix2, offset2, destination_cs=global_cs)
   
   # Automatically join together [image2.transform.inverse, image1.transform]
   composed_transform = image1.pixel_cs.transform_to(image2.pixel_cs)
   
   # Map pixel location from one image to another
   pixel_inds2 = composed_transform.map(pixel_inds1)
   
   # Alternatively, these could be combined into one line:
   pixel_inds2 = image1.pixel_cs.map_to(image2.pixel_cs, pixel_inds1)

5. One issue not resolved is how to reference external standardized coordinate systems; something like:

   ccf_pos = image.pixel_cs.map_to("AICCF", pixel_inds)

   Most of the python transform infrastructure is being developed as an independent package: https://github.com/campagnola/transformy

[bendichter]

@DavidTingley @samamckenzie

[rly]

We're closing this issue for now, but it is closely related to NWB-related ontologies work being done by @lydiang.