NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files
BSD 2-Clause "Simplified" License
49 stars 32 forks source link

Unable to read allen instititute visual coding files #224

Closed donghoon-shine closed 1 year ago

donghoon-shine commented 4 years ago

Hi team NWB!

I am trying to use open-source Neuropixel recording in .nwb extention from Allen Insititute. ( https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels ).

Since all my existing codes are in matlab I'd like to use matlab to process the file. However, I am not able to read files with matnwb. These is the error message I get with matnwb:

Error using types.util.checkUnset (line 13) Unexpected properties {help}.

Your schema version may be incompatible with the file. Consider checking the schema version of the file with util.getSchemaVersion(filename) and comparing with the YAML namespace version present in nwb-schema/core/nwb.namespace.yaml

Error in types.core.TimeSeries (line 82) types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseGroup (line 85) parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33) nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Error in nwbprocess_script1 (line 6) nwb2 = nwbRead(filename);

When I type 'util.getSchemaVersion(filename)', i get {'2.0b'}. My YAML namespace version present in nwb-schema/core/nwb.namespace.yaml is 2.2.4.

Please help me with this problem.

Thank you! Donghoon

cspurandare commented 4 years ago

Hello I am facing the same issue as well. Looking forward to some way forward, Thanks, Chinmay

lawrence-mbf commented 4 years ago

(I apologize for not getting to this issue Friday, it got caught by my spam folder) The schema in MatNWB is, by default, from schema version 2.2.4 which is the most recent schema version.

You will need to generate new classes based on an older schema that is compatible with 2.0b. The closest schema I was able to find is this version: https://github.com/NeurodataWithoutBorders/nwb-schema/releases/tag/2.0.1 which did appear to work with other users. The instructions for installation are below:

  1. Download and unzip the source code in the link above.
  2. In the unzipped directory, go to the core folder and find a file called nwb.namespace.yaml
  3. In MATLAB call generateCore with the full path to the nwb.namespace.yaml file found above.
  4. Try and read from the file again. Note that the class and cache files (+types and namespaces directories) will be created in your current MATLAB workspace).
bendichter commented 4 years ago

Yes, you will need to make sure you are using the right schema. Let us know if you have any more problems.

cspurandare commented 4 years ago

Hi, Thank you for your help. I was able to fix the schema compatibility issue with the steps 1-4 above. But now I am facing an error about AIBS_ecephys.EcephysProbe. This reads as

nwbRead('C:\ChinmayStuff\AllenNeuropixelStuff\ecephys_session_715093703.nwb') Undefined variable "types" or class "types.AIBS_ecephys.EcephysProbe".

Error in io.parseGroup (line 85) parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33) nwb = io.parseGroup(filename, h5info(filename), Blacklist);

I checked in the +types folder and there is no folder for AIBS_ecephys, which seems to get created for newer .nwb files and schemas(eg. 2.1.0) I have tried using a slightly newer schema than 2.0.1 which I am currently using, but that gives the incompatibility error again. I have tried to copy the AIBS_ecephys folder from some other +types folder but that gives the error and NWBContainer has a superclass to define AIBS_ecephys, so I dont think that the hack is appropriate. I am using a windows machine, if that changes anything? Thanks again, Chinmay

lawrence-mbf commented 4 years ago

Does the error message change if you copy the AIBS_ecephys folder from the different directory?

AIBS_ecephys is an extension to the regular NWB classes so you'll need to generate those classes as well since they're also not embedded into the NWB file. I do not know where you can download that extension schema, but when you do find them, you just follow the instructions I laid out above for the core schema but replacing the generateCore call with generateExtension instead. Everything else should be the same.

cspurandare commented 4 years ago

I dont think I explicitly copied those yamls but they got created when I was trying to open some files from Steinmetz Nature paper using the 2.1.0 schema. I tried to run generateExtension therein, but got a Java error. When I try to copy the resultant folder from their +types into the +types of the installation with 2.0.1 schema, my error changes to

Error using types.AIBS_ecephys.EcephysProbe Cannot define property 'help' in class 'EcephysProbe' because the property has already been defined in the superclass 'NWBContainer'.

Error in io.parseGroup (line 85) parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38) subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33) nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Does this information help? Or am I missing some steps or messed up sequence of something? Chinmay

lawrence-mbf commented 4 years ago

By the Steinmetz Nature do you mean the one here? https://www.nwb.org/example-datasets/ The datasets I've looked at there have embedded specifications so a nwbRead should work as normal without generating anything.

lawrence-mbf commented 4 years ago

Ah, I understand, the files were generated using the schema embedded in the Steinmetz paper. There's no guarantee that AIBS_ecephys is the same version between the two files considering they're using vastly different NWB schemas.

The ones for Steinmetz have assumed NWB schema version 2.1.0 which may not be true for the Allen dataset.

cspurandare commented 4 years ago

Yes it was from that website. I agree, that the differences in the NWB schemas are causing the non-compatibility for ecephys files. I actually found a work around. I downloaded the HDFView 3.3.1 from

https://www.hdfgroup.org/downloads/hdfview/

Now I can explore the contents of the nwb using that viewer and upon identification of the "subfields" in the nwb file of interest, I can import necessary fields with the h5read function in MATLAB. This is slightly tedious, since now I require 2 different programs, MATLAB and that viewer, but I am imagining that the nwbRead function has underlying dependencies on that h5read or equivalent reader anyway. Do you think this would be sustainable or do you foresee issues I havent encountered yet? I came to know of this viewer (https://smackesey.github.io/nwb_tutorial/assets/tutorial.html) and h5read( from https://github.com/SteinmetzLab/dataToNWB) from these sources. Chinmay

lawrence-mbf commented 4 years ago

@bendichter Do you know anything about this version of the AIBS_ecephys extension?

@cspurandare I would recommend finding a schema that works, there are a few HDF5 types that are not handled properly by h5read, namely when it comes to reference data. You'll have to use the lower level H5R functions in MATLAB to properly handle those: https://www.mathworks.com/help/matlab/reference-h5r.html

lawrence-mbf commented 1 year ago

Closed as stale and most NWB files should have an embedded spec now.