error with generateCore() after latest push

[mpompolas]

I have set up Brainstorm to pull the latest version from GitHub when doing a clean install.

After the latest push I get this:

generateCore() Error using generateCore (line 36) Cannot find suitable core namespace for schema version 2.2.5

[babiec]

Same thing happened to me. Loaded new push with R2020b and got the same error message following generateCore().

Error using generateCore (line 36) Cannot find suitable core namespace for schema version 2.2.5

[lawrence-mbf]

Unfortunately, this was due to the inclusion of submodules in the new release, which git pull does not properly update on its own. You will need to call git pull --recurse-submodules. If that doesn't work, you're using an older version of git and should use git submodule update --init --recursive instead.

[mpompolas]

unfortunately this is not an option from my side. I can't expect Brainstorm users to have git. Is it possible to add a flag for loading a cached version from a local directory as well on generateCore? (I'm still trying to make it work, but I will probably cache the main NWB releases on the Brainstorm side)

[bendichter]

Does the zipped version of the repo available for download on GitHub contain subrepos?

On Thu, Dec 24, 2020, 4:25 PM Konstantinos notifications@github.com wrote:

unfortunately this is not an option from my side. I can't expect Brainstorm users to have git. Is it possible to add a flag for loading a cached version from a local directory as well on generateCore? (I'm still trying to make it work, but I will probably cache the main NWB releases on the Brainstorm side)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NeurodataWithoutBorders/matnwb/issues/251#issuecomment-751063068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGOEEQBU7JYUHR33XZGVBDSWOWVTANCNFSM4VIOO3KA .

[lawrence-mbf]

unfortunately this is not an option from my side. I can't expect Brainstorm users to have git. Is it possible to add a flag for loading a cached version from a local directory as well on generateCore? (I'm still trying to make it work, but I will probably cache the main NWB releases on the Brainstorm side)

generateExtension doesn't actually care what namespace file you provide it, so you could use that and feed it local core namespace files instead of using generateCore. Just keep in mind that the current core namespace is dependent on the hdmf-common namespace.

Does the zipped version of the repo available for download on GitHub contain subrepos?

Just checked, it does not so you need to use git to download the submodules.

[bendichter]

Maybe it would make sense for us to provide a downloadable package of all of the hdmf-common and pynwb releases somewhere for Brainstorm and other systems that cannot expect git to be available.

[lawrence-mbf]

You could just use the released assets in nwb-schema and hdmf-common directly though that would require a separate script. Doing it this way also requires special logic for version 2.2.3 because its release version is using broken YAML, you'll have to download the schema file from the commit that fixes it.

[lawrence-mbf]

Another option is to ditch submodules entirely. I'm not entirely sure what benefit actually using submodules provide other than git versioning support.

[bendichter]

Yeah, when I suggested it I didn't realize that Brainstorm wouldn't be able to access older repos because it can't rely on git being present. The space savings vs. storing every schema version is minimal. I'd be fine with just checking all released versions into MatNWB

[mpompolas]

I believe the problem is also a bit deeper - I tried the past few days to make a transition between Schema versions (OUTSIDE of Brainstorm - never loaded it) and I wasn't able to make it work. Tried both just storing the schemas and also storing the entire MatNWB versions and calling generateCore.

I think we should make a script so we can all take a look at the steps needed for the transition between schemas for reproducibility of the results. I can create one where it also downloads files from DANDI so we use the same files.

[lawrence-mbf]

I'm not entirely sure what you mean by it not working. Were you using the most recent matNWB version? What error messages do you get? How are you calling generateCore?

I think I forgot to mention this in our call but generateCore by default uses a hardcoded full filepath. In previous editions of MatNWB you could add custom file paths to namespace.yaml files in order to generate extension classses off of a single call (this basically called generateExtension on all the paths). With the new release this functionality was removed and generateExtension should be used to generate off of custom YAML file locations.

[mpompolas]

two approaches:

1. 2.2.5.0 MatNWB (before your submodule insertion). Had just the schemas of each version saved on separate folders. When I wanted to load a file, I would read its version, put the associated schema folder on top of the PATH and then load the file.
2. Downloaded all released MatNWB versions (2.2.1 up to 2.2.5.0). Read the version of the file. Add the associated MatNWB on top of the path, call generateCore() from within the directory (no need for a filepath) and then call readNwb().

The most common issue was an incompatible schema error (I really don't understand why it was throwing that - maybe my approach was wrong).

Should I scratch everything and just use 2.2.5.1?

I think a script that showcases how to transition between the versions is the way to go. I will start preparing it and you can edit as well.

[lawrence-mbf]

So if you're trying to avoid loading the Schema java object, you should skip the namespace step entirely. I'm assuming you're looking for a script that auto-generates the class files into a specific directory?

In any case, the YAML files do not depend on a PATH, only the class files (+types/.../*.m) files do. The YAML files are loaded by full path. My assumption was that you'd be using the PATH to specify the generated class files, not generating through the YAML files.

[mpompolas]

I'm assuming you're looking for a script that auto-generates the class files into a specific directory?

I think that would do the trick. Right now I am jumping between folders depending on the version within the File. Which is fine, but not all files I'm testing work. It might just be inconsistencies between the releases and the DANDI files that is causing my headaches. For example one error I just came across:

DANDI dataset: https://girder.dandiarchive.org/api/v1/item/5eda806c99f25d97bd279816/download MatNWBRelease: https://github.com/NeurodataWithoutBorders/matnwb/archive/0.2.3.zip

In class 'types.untyped.DataStub', no set method is defined for Dependent property 'dims'.  A Dependent
property needs a set method to assign its value.

Error in types.untyped.DataStub (line 16)
            obj.dims = fliplr(h5_dims);

Error in io.parseDataset (line 64)
        data = types.untyped.DataStub(filename, fullpath);

Error in io.parseGroup (line 22)
    dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Error in schema_checks (line 93)
    nwb2 = nwbRead(outFileNames{iFile});

[mpompolas]

I added a script (schema_transition_checks.m) to download all needed NWB and schema files and also try to transition between versions in this fork: https://github.com/mpompolas/matnwb/tree/check_schemas

This way we can hopefully reproduce all these issues

[lawrence-mbf]

Oh you're downloading MatNWB versions? I would actually advise downloading the nwb-schemas directly. There are definitely bugs fixed in later versions that would apply to older MatNWB versions. For version 2, MatNWB 2.2.5.1 should be compatible with all schema versions 2 and above (hence the addition of the version switcher). I think it will be easier for you if we actually removed the submodules and simply embedded the nwb schema by version. That way, it will be much easier for you to generate schema versions depending on what's needed.

[mpompolas]

There are definitely bugs fixed in later versions that would apply to older MatNWB versions.

ooooooohhhh! This is what probably was driving me nuts. I'll give it a try now by downloading only the schemas as you suggested and try it with 2.2.5.0 (2.2.5.1 doesn't work since it needs the submodules).

I think it will be easier for you if we actually removed the submodules and simply embedded the nwb schema by version. That way, it will be much easier for you to generate schema versions depending on what's needed.

In the long run, I think that would be the most efficient solution so it doesn't need maintenance from the Brainstorm side. Keep me posted of the developments

[mpompolas]

updated the code on the script (schema_transition_checks.m) to use 2.2.5.0 and download all schemas now: https://github.com/mpompolas/matnwb/tree/check_schemas

still getting an error though on the second file (version 2.2.2) while running the script. Maybe something is wrong with the ordering of the folders I add to the path, maybe the file is problematic - I can't know for sure. Can someone try to run this script and give some feedback please? On the other hand, if there is a plan in the near future to:

simply embedded the nwb schema by version

as @ln-vidrio mentioned, I will just wait for that to be implemented.

Error reported:

Unrecognized property 'vectorindex' for class 'types.hdmf_common.DynamicTable'.

Error in types.hdmf_common.DynamicTable (line 26)
        [obj.vectorindex, ivarargin] = types.util.parseConstrained(obj,'vectorindex',
        'types.hdmf_common.VectorIndex', varargin{:});

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Error in schema_transition_checks (line 95)
    nwb2 = nwbRead(outFileNames{iFile});

>> 

[lawrence-mbf]

Try v2.2.5.3, it no longer uses submodules.

EDIT: v2.2.5.2 did not convert schema 2.2.5 correctly. use v2.2.5.3 instead.

[mpompolas]

Thank you very much @In-vidrio.I'm taking a few days off, so I will work on this again next weekOn Dec. 30, 2020 11:17, ln-vidrio notifications@github.com wrote: Try v2.2.5.2, it no longer uses submodules.

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

[mpompolas]

hey @In-vidrio. Happy new year. The changes you made with v2.2.5.3 seem to be working for transitioning between versions. I removed the schema-caching from the Brainstorm side.

Tried on several files from DANDI and everything seems good so far.

I'll let you know if I find any exotic datasets that create problems

[lawrence-mbf]

@babiec, this appears to work for @mpompolas so I will be closing this ticket. If you have further questions, submit a new one or, if it is directly related to generateCore, re-open this ticket.

[babiec]

Thank you for all of the effort!

On Wed, Jan 27, 2021 at 7:42 AM ln-vidrio notifications@github.com wrote:

@babiec https://github.com/babiec, this appears to work for @mpompolas https://github.com/mpompolas so I will be closing this ticket. If you have further questions, submit a new one or, if it is directly related to generateCore, re-open this ticket.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeurodataWithoutBorders/matnwb/issues/251#issuecomment-768373713, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHJAC7YGIPVUOYDFX6UUGLS4AX6RANCNFSM4VIOO3KA .

-- Lecturer, Undergraduate Interdepartmental Program for Neuroscience College of Life Sciences University of California, Los Angeles