search

NeurodataWithoutBorders / pynwb

A Python API for working with Neurodata stored in the NWB Format
https://pynwb.readthedocs.io
Other
175 stars 85 forks source link

nwb-schema and hdmf-common-schema compatibility #1289

Open rly opened 4 years ago

rly commented 4 years ago

Each release of the NWB schema is bundled with a particular release of the hdmf-common schema. Each release of HDMF is bundled with a particular release of the hdmf-common schema. Each release of PyNWB is bundled with a particular release of the NWB schema and is compatible with a range of releases of HDMF. This can lead to a mismatch between the version of hdmf-common schema included in nwb-schema and that used by PyNWB.

For example: PyNWB version 1.4.0 is bundled with NWB schema 2.2.5 which uses hdmf-common schema 1.1.3. This version of PyNWB uses HDMF >=2.1.0, <3. HDMF 2.1.0 is bundled with hdmf-common-schema 1.2.0. PyNWB will load whichever version of hdmf-common schema is used by the installed version of HDMF.

So a file generated using PyNWB version 1.4.0 will cache NWB schema 2.2.5 and newer hdmf-common schema 1.2.0.

For now, this is fine, but I can imagine a breaking change in hdmf-common schema bumping it to 2.0.0, such that an earlier NWB schema is not compatible with it.

Proposed solution: 1) The schema language should be updated so that a namespace version can (and must?) be specified by a namespace dependency. 2) PyNWB should raise an error when loading namespaces that are incompatible with each other (consider extensions) 3) PyNWB should pin to a particular release of HDMF, not a range. 4) The NWB schema version used by PyNWB must use the same version of hdmf-common schema as the pinned release of HDMF.

oruebel commented 4 years ago

Sounds good. I believe (1) is already supported to allow specify namespace versions (but I don't think there is any error checking done yet).

rly commented 4 years ago

@oruebel unless I am not looking in the right place, I don't see (1) here: https://schema-language.readthedocs.io/en/latest/specification_language_description.html#schema

To be clear, I mean that this line https://github.com/NeurodataWithoutBorders/nwb-schema/blob/dev/core/nwb.namespace.yaml#L20 should include a version

oruebel commented 4 years ago

I thought the spec classes in HDMF allow this already, but I may be wrong.

rly commented 4 years ago

@oruebel I just read through hdmf/spec/namespace.py where other parts of namespaces > schema are parsed, and I see no reference to versions of dependent namespaces. I'll propose a change to the language and HDMF to account for this.

oruebel commented 4 years ago

I'll propose a change to the language and HDMF to account for this.

Sounds good.

rly commented 4 years ago
  • PyNWB should pin to a particular release of HDMF, not a range.

Revision: Instead of the above, if HDMF is updated to support a new version of hdmf-common-schema, then we increase the major version number in HDMF. This way, PyNWB can continue using a range within the major version number and users can continue to receive non-schema related bug fixes. Changes to hdmf-common-schema should be relatively low frequency now anyway.

We could also adopt a similar strategy with PyNWB of bumping the major version number whenever PyNWB supports a new version of nwb-schema.