Wishlist/Recommendation: "Downstream testing" against NWB-Extensions

[yarikoptic]

NWB extensions have their lives, and NWB-schema/HDMF/PyNWB and those extensions as well, can progress forward while possibly missing the point whenever a change in any of those caused some files containing the extensions not even loadable (#1337, #1335).

To help (if not to eliminate) to identify and fix similar issues early in the process, and to make development of pynwb aligned with the ecosystem of extensions it would be great if per each extension there were

• sample nwb files with a given extension, possibly multiple files to correspond to various versions of the extension/base pynwb which produced them

and have a setup where

• all extensions are tested against current released as well as against "in development" hdmf/PyNWB
• rudimentary smoke (or + some basic unittests for expected data structures etc) test of PyNWB to load sample files mentioned above

PS feel free to refile somewhere against extensions, but as pynwb in my view is the entry point for users, it should also be pynwb's "duty" to CI against extensions on regular basis if not in each PR. Eg. in datalad we do test against only a limited subset of extensions within PRs (https://github.com/datalad/datalad/actions?query=workflow%3AExtensions) and then across all of the extensions in a dedicated repo/dashboard: https://github.com/datalad/datalad-extensions/#datalad-extensions-status . May be some of those ideas/setups come handy (datalad-extensions setup should be quite generalizable - workflows are templated from a single spec)

[oruebel]

I agree that having testing infrastructure for extensions will be useful. I'm not sure, whether this will need to be part of the extension catalog, individual extensions, or PyNWB (probably some combination of things). I thin this would be a good topic for the upcoming hackathon to do some specification on.

[sneakers-the-rat]

bumping this as i was searching issues for a PR and have been wanting this as well - so far my strategy has been to mirror the entirety of dandi and do roundtrip tests, which is fun, but it would probably be more efficient to have smoke as per OP. in reimplementing the standard i think i've started to arrive at a pretty succinct and separable set of cases with logical conditions that could be inverted into fuzz, been trying to mark them as i go, but curious about the status of 'edge case test case' datasets and the like.

i am seeing the backwards compat files now which i'm gonna adopt - i've rigged up providers that can generate linkml -> pydantic cached across any version from the git repo (and just provided prebuilt) as part of a lossless translation where schema extensions should work as well, so i would be happy to contribute to a test and validation CI action/suite that could be tested against all versions of extensions