io.read() gives error on Allen NWB files

[INF-Rene]

1) Bug

When downloading NWB files through the AllenSDK I can't seem to read the files using the io.read() command after creating an NWBHDF5IO object. When I open the file with an external NWB reader there is no problem.

Steps to Reproduce

I have obtained the nwb file using this code:

from allensdk.core.cell_types_cache import CellTypesCache
from allensdk.api.queries.cell_types_api import CellTypesApi
from allensdk.core.cell_types_cache import ReporterStatus as RS
from allensdk.ephys.ephys_extractor import EphysCellFeatureExtractor
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

ctc = CellTypesCache(manifest_file='cell_types/manifest.json')

cells = ctc.get_cells(species=[CellTypesApi.HUMAN])
print("Human cells: %d" % len(cells))

data_set = ctc.get_ephys_data(cells[1]['id'])

Then I navigate to the specimen folder where ephys.nwb is located and run the following code:

from datetime import datetime
from pynwb import NWBFile, NWBHDF5IO
io=NWBHDF5IO('ephys.nwb', mode='r')
nwbfile = io.read()

Giving me the following error:

C:\Anaconda\lib\site-packages\pynwb\form\build\builders.py:211: UserWarning: 'comments' already exists as attributes; skipping...
  warnings.warn("'%s' already exists as %s; skipping..." % (name, self.obj_type[name]))
Traceback (most recent call last):

  File "<ipython-input-4-23327a8afbc1>", line 2, in <module>
    nwbfile = io.read()

  File "C:\Anaconda\lib\site-packages\pynwb\form\utils.py", line 337, in func_call
    return func(self, **parsed['args'])

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\io.py", line 32, in read
    f_builder = self.read_builder()

  File "C:\Anaconda\lib\site-packages\pynwb\form\utils.py", line 337, in func_call
    return func(self, **parsed['args'])

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\hdf5\h5tools.py", line 217, in read_builder
    f_builder = self.__read_group(self.__file, ROOT_NAME, ignore=ignore)

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\hdf5\h5tools.py", line 289, in __read_group
    builder = read_method(sub_h5obj)

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\hdf5\h5tools.py", line 289, in __read_group
    builder = read_method(sub_h5obj)

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\hdf5\h5tools.py", line 289, in __read_group
    builder = read_method(sub_h5obj)

  File "C:\Anaconda\lib\site-packages\pynwb\form\backends\hdf5\h5tools.py", line 297, in __read_group
    ret = GroupBuilder(name, **kwargs)

  File "C:\Anaconda\lib\site-packages\pynwb\form\utils.py", line 337, in func_call
    return func(self, **parsed['args'])

  File "C:\Anaconda\lib\site-packages\pynwb\form\build\builders.py", line 149, in __init__
    self.set_dataset(dataset)

  File "C:\Anaconda\lib\site-packages\pynwb\form\utils.py", line 337, in func_call
    return func(self, **parsed['args'])

  File "C:\Anaconda\lib\site-packages\pynwb\form\build\builders.py", line 244, in set_dataset
    self.__set_builder(builder, GroupBuilder.__dataset)

  File "C:\Anaconda\lib\site-packages\pynwb\form\build\builders.py", line 214, in __set_builder
    (name, self.obj_type[name], obj_type))

KeyError: "'description' already exists as attributes, cannot set as datasets"

Environment

Please describe your environment according to the following bullet points.

• Python Executable: Python.exe
• Python Version: Python 3.6
• Operating System: Windows
• Pynwb Version: 0.5.1

Edit: updated from 0.4.1 to 0.5.1, same result

[oruebel]

@nicain can you respond to this issue?

[nicain]

Thanks for the report, @INF-Rene! Can you please give more detail about the external NWB reader that works successfully? I am guessing that this is a NWB 1.x vs 2.0 issue, but I want to find the best way to route this information so that it can be addressed. CC @sgratiy @dyf

[INF-Rene]

Thanks @nicain ! I'm sorry to say I was completely wrong about the external viewer. I mixed up with another file. I used HDFview 3.0.0, but the viewer said the file format is unsupported.

When I open the file in notepad I can see the file version in between the jibberish, which is NWB-1.0.5.

Hope this helps. If you want to try on the file you can download the file using the code in my opening post, but if you prefer I can upload it.

[INF-Rene]

I just tried to open an NWB 2.0 file from the Allen institute and it gave me no problems. Is there any method to bulk convert NWB 1.x to 2.0 ?

[t-b]

@INF-Rene Not that I'm aware of.

[nicain]

@JFPerkins is working on some tooling to handle this for AIBS Brain Observatory files. Will that cover your needs? CC @sgratiy

[INF-Rene]

That would be perfect! Thank you and let me know if there is anything I can do to help

[sarmishta]

@INF-Rene I am facing the same issue, please could you tell me how did you solve it? Thank you.

[rly]

Hi @sarmishta Just to be clear, which file from the Allen Institute are you having trouble loading? If this is an intracellular recording stored in NWB 1.0, you can use the resources below:

This tutorial may be useful for reading the NWB 1.0 data: https://smackesey.github.io/nwb_tutorial/assets/tutorial.html

There is also documentation about the NWB 1.0 format at: https://github.com/NeurodataWithoutBorders/specification_nwbn_1_0_x/tree/master/version_1.0.6 And at: http://neurodatawithoutborders.github.io/api-python/build/html/index.html

[INF-Rene]

Hi @sarmishta

Unfortunately NWB files generated by allen institute or by their acquisition software (see https://github.com/AllenInstitute/MIES) are NWB v1 and as far as I know still not working in either pynwb or matnwb.

It's really a pity since me and many electrophysiologists around me are switching to MIES for acquisition.

Currently I am recovering the data using my own scripts in MATLAB, which is just using the hdf5read function which comes built-in with MATLAB. Python has a similar package.

Of course this is far from ideal, but for me it was the only viable solution for working with NWB v1 files at the moment.

Good luck!

[t-b]

@INF-Rene We are already working on the NWBv2 support. Feel free to subscribe to https://github.com/AllenInstitute/MIES/issues/22 to get notified when its done.

[sarmishta]

@INF-Rene,@rly Thank you so much for the reply. We were looking at the Allen cell Type Database(https://celltypes.brain-map.org/experiment/electrophysiology/528706755 ) , to extract spike features from the AllenSDK using CellTypesCache Class. We noticed that the threshold_t spike feature does not match the cell response X-Aixis I.e time in seconds. Our guess is that, the start time of the stimulus and response for all the cells were modified by an offset for them to converge at the same time. Do you have any idea on what is the offset value? Thank you!

[t-b]

I'm closing this issue as read support for NWBv1 files is not planned to be part of pynwb.

Feel free to reopen if someone wants to work on that.