Variable-shape ROIs

[ageorgou]

Currently the code assumes that all ROIs have the same shape. The main complication from allowing the shape to vary is storing the pixel time offsets. At the moment we are storing them in the plane segmentations, which is a DynamicTable. Adding two ROIs (on the same plane) with different shapes for the pixel time offset array causes an error.

Potential solutions:

• ❌ Flattening the time offsets to a one-dimensional array for each ROI (same problem)
• ❌ Using H5DataIO to wrap each ROI's offsets when adding them (the wrapper in principle allows for resizeable arrays using maxshape, but it doesn't seem to solve the problem)
• ❓ Gathering all the offsets in a single data structure (as yet unspecified...) and adding them all at once, which DynamicTable allows (putting them in a list doesn't work, even with the H5DataIO wrapping, but there may be other ways such as HDF5 variable length types)
• ❓ Changing how we create the pixel_time_offsets column in the DynamicTable so that it allows ragged arrays, which its docs mention
• ❓ Moving the time offsets somewhere else, such as to our custom optophysiology data, where we might have more flexibility

(If the shape of ROIs only varies across different planes, and is consistent within each plane, then the current approach should work fine, but I'm assuming that is not generally the case)

Additionally, the resolution (pixel size) may change across ROIs, so we should make sure we either store that or adapt the code to take accordingly.

[alessandrofelder]

this example code might serve as a starting point in the future (next week) to implement the second ❓ option above:

import datetime
from pynwb import NWBHDF5IO, NWBFile
from pynwb.ophys import OpticalChannel, ImagingPlane, ImageSegmentation

# create file
file = NWBFile('bla.nwb', identifier='bla identifier', session_start_time=datetime.datetime(1999, 5, 5))
file.create_device('an empty device', 'empty device description')

# set up pynwb imaging plane and segmentation
imaging_plane = file.create_imaging_plane(name='an imaging plane',
                                          optical_channel=OpticalChannel('red', 'red channel description', 345.5),
                                          description='this text describes the imaging plane',
                                          device=file.devices['an empty device'],
                                          excitation_lambda=122.0,
                                          imaging_rate=2.0,
                                          indicator='Nimbus2000',
                                          location='hypothalamus')

image_segmentation = ImageSegmentation()
plane_segmentation = image_segmentation.create_plane_segmentation(
    description='this text describes the plane segmentation',
    imaging_plane=imaging_plane,
    name='plane seg')

# need to add empty rows and then index the pixel offsets in the added column
plane_segmentation.add_row()
plane_segmentation.add_row()
plane_segmentation.add_column(name='pixel_time_offsets',
                              description='this text describes the pixel time offsets',
                              data=[1, 2, 3, 4, 5],
                              index=[2, 5])

module = file.create_processing_module(
    'Acquired_ROIs',
    'ROI locations and acquired fluorescence readings made directly by the AOL microscope.')
module.add(image_segmentation)

# write the file
with NWBHDF5IO('example_nwb_file_with_ragged_pixel_time_offsets.nwb', 'w') as io:
    io.write(file)

[ageorgou]

Will look at the following approaches:

1. Store pixel time data in a long 1 dimensional array, using a VectorIndex to break it down by ROI. Since we have the shape of each ROI, we can put the 1-d chunk in the right shape (provide custom methods for that).
2. Store dummy data of a dummy shape when creating the ROIs, then change it manually using h5py.
3. Define a custom type for a ROI's pixel time offsets, create it on the file somewhere, and store a reference to it in the pixel_time_offsets ROI Table.

[alessandrofelder]

Exploring option 3 of the latest comment, and understanding that the ROI data is stored as an ImageSeries under acquisition, we now have a minimal example that proves the concept. A sketch of it is here, so we remember.

First, we extend the schema with a new class derived from ImageSeries that contains an additional dataset for the pixel_time_offsets:

   # dimensions ordered as t, x, y [, z], like the TimeSeries data itself
    silverlab_pixel_time_offset_data = NWBDatasetSpec(doc='A datastructure to hold time offsets for pixels. The'
                                                          'time offsets are the acquisition time of each pixel '
                                                          'relative to the starting time/timestamp of a '
                                                          'ROISeriesWithPixelTimeOffsets.',
                                                      name='pixel_time_offsets',
                                                      shape=[(None, None, None), (None, None, None, None)],
                                                      neurodata_type_def='PixelTimeOffsets')

    silverlab_roi_image_specs = NWBGroupSpec(doc='documentation of this class goes here',
                                             datasets=[silverlab_pixel_time_offset_data],
                                             neurodata_type_def='ROISeriesWithPixelTimeOffsets',
                                             neurodata_type_inc='ImageSeries')

This allows to create the ROI images and connect them to the pixel_time_offsets:

PixelTimeOffsets = get_class('PixelTimeOffsets', 'silverlab_extended_schema')
ROISeriesWithPixelTimeOffsets = get_class('ROISeriesWithPixelTimeOffsets', 'silverlab_extended_schema')

pixel_time_offsets = PixelTimeOffsets(
    data=...)

assert (np.shape(xy_plane_image_time_series) == np.shape(pixel_time_offsets))
roi_with_pixel_time_offsets = ROISeriesWithPixelTimeOffsets(name='ROI_0',
                                                            data=image_time_series,
                                                            pixel_time_offsets=pixel_time_offsets,
                                                            unit='some unit',
                                                            timestamps=[0., 1., 2.])

There may be able to improve on this by deriving from RoiResponseSeries instead of ImageSeries, as that should allow us to additionally link the ROI image data to the ROI Table.