Enable write of compound data types with FORM

[ajtritt]

Users would like to specify, read, and write compound data types.

The following subtasks will be needed to accomplish this.

• [x] #77
• [x] #78

[nicain]

First cut at this in: https://github.com/nicain/pynwb/tree/feature/table_writer

Try with this sample code:

import numpy as np
from pynwb import get_class, load_namespaces
from datetime import datetime
from pynwb import NWBFile
import pandas as pd
from form.backends.hdf5 import HDF5IO
from pynwb import get_build_manager

ns_path = "alleninstitute.namespace.yaml"
ext_source = "alleninstitute.extensions.yaml"

help_attr = NWBAttributeSpec(name='help',
                             dtype='str',
                             doc='no help for you',
                             value='YUP')

nwb_ds = NWBDatasetSpec(doc='dataset_doc',
                        dtype=[{'dtype': 'int', 'label': 'foo'},
                               {'dtype': 'float', 'label': 'bar'}],
                        name='table')

ns_builder = NWBNamespaceBuilder('Allen Institute extensions', "alleninstitute")
ext = NWBGroupSpec('An example table spec',
                   datasets=[nwb_ds],
                   neurodata_type_inc='NWBContainer',
                   neurodata_type_def='Table',
                   attributes=[help_attr])

ns_builder.add_spec(ext_source, ext)
ns_builder.export(ns_path)

ns_path = "alleninstitute.namespace.yaml"
load_namespaces(ns_path)

Table = get_class('Table', 'alleninstitute')
fs = Table(table={'foo':[0, 1, 2], 'bar':[3, 4, 5]},
           name='my_population',
           source='acquisition',
           unit='second',
           help='None')

f = NWBFile(file_name='tmp.nwb',
            source='me',
            session_description='my first synthetic recording',
            identifier='EXAMPLE_ID',
            session_start_time=datetime.now(),
            experimenter='Dr. Bilbo Baggins',
            lab='Bag End Labatory',
            institution='University of Middle Earth at the Shire',
            experiment_description='empty',
            session_id='LONELYMTN')

f.add_stimulus(fs)

manager = get_build_manager()
io = HDF5IO('tmp.nwb', manager, mode='w')
io.write(f)
io.close()

[oruebel]

A couple of quick questions and comments for discussion:

• Based on the discussion in https://github.com/NeurodataWithoutBorders/nwb-schema/issues/13 I believe we also want to add a 'doc' key for each component of the data type
• Instead of using str and list to define dtypes, I think it would be useful to add a new NWBDataTypeSpec class for handling specifications of dtypes, to make the concept easier to use, change, and validate.
• How do we want to represent data tables for read and write? From an h5py perspective and since we are already using numpy, it seems that using structured arrays (i.e., arrays with structured dtypes) would make sense ( https://docs.scipy.org/doc/numpy-1.10.1/user/basics.rec.html ). This may also help simplify the read/write portion a bit. To help user, one could even add a simple function to create the structured dtype from a spec (e.g., as a function on the NWBDataTypeSpec) and/or as a helper method on the corresponding interface classes in PyNWB.
• Do we also need to allow compound types for attributes?

[ajtritt]

Thanks for the feedback Oliver. Here are my thoughts to the issues you have raised (in the same order)

• I agree, we should require documentation for each column.
• I have had the same thought about creating a spec class for compound data types, but I think it needs to be done in FORM, and therefore called something like DataTypeSpec. We can add NWBDataTypeSpec to pynwb.spec after that the general class is completed.
• I think we can start with numpy structured arrays, but we should eventually support xarray.DataArray and pandas.DataFrame. As @nicain has already done, we can support a dict of lists (or numpy arrays).
• As of now, I do not know of a use case for compound types for attributes. I think we should focus just datasets for now, since it seems to be in higher demand, as well as holding up other issues

[ajtritt]

Example use of table/compound data type from @nicain https://github.com/nicain/example_nwb

[oruebel]

@ajtritt I assume this issue can be closed as well?

[jcfr]

Based on most recent comment. Closing.

Please, re-open if this is a mistake.