search

rly / linkml-arrays-paper

Perspective Paper on LinkML Arrays
0 stars 0 forks source link

Initial high-level goals and outline #2

Open rly opened 3 months ago

rly commented 3 months ago

Goal: A slightly technical perspective piece describing the problem of representing arrays in schema and how array support within LinkML solves the problem

A rough first outline:

  1. Background
    • The historical challenges of representing arrays in schema with rich metadata (e.g., linkages to ontologies)
      • Previous work, including HDMF, CORAL, the NeXus data format for X-ray and neutron scattering and muon spectroscopy, netCDF?, OME-NGFF, geospatial data?, JSON schema arrays, previous approaches in LinkML
      • Many file formats for arrays - binary file, numpy npy/npz, HDF5, Zarr, N5, JSON, CSV/TSV, grib, tiff, fits
      • Many APIs for working with these formats - numpy, h5py, zarr, xarray, etc.
    • Need a unifying framework across these technologies
      • What is LinkML
  2. Adding array support in LinkML - some technical details
    • Specification of simple and labeled arrays
    • Generation of Pydantic API for arrays
    • Dumpers and loaders from/to various file formats from/to various (python-based) APIs
    • Validation
  3. Examples in at least two applications
    • Neurophysiology
    • Bioimaging
    • Environmental / geospatial
  4. Discussion
    • Challenges of open data in science
    • Expanding coverage / use cases of LinkML

Potential target journals:

We would write a second paper on LinkML arrays for NWB/neurophysiology, more for a neuroscience audience.

cmungall commented 2 months ago

https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/5.1.0/schema.md

oruebel commented 2 months ago

Just some links to a few other array-based file formats that may be of interest:

sneakers-the-rat commented 2 months ago

OK am back from vacation and ready to rumble.

made some child issues off this one to start tracking different pieces and make 'threads' for discussing those sub-points

sneakers-the-rat commented 2 months ago

Sry have been preoccupied with events on campus, will be returning to this next week. Numpydantic is near a 1.0 except for tests and docs, so its Numpydantic 1.0 -> new array range generator -> update nwb-linkml to reflect.

The intro to the paper can happen async, but once we have those pieces in place we can do the meat of the results

sneakers-the-rat commented 1 month ago

hey every1. i just "officially" released numpydantic, so the next step now is to put that in linkml arrays generator (should only take ~a day) and then rework nwb-linkml (a ~week) before starting the paper.

unfortunately, my employer has decided to commit egregious unfair labor practices in the form of police violence against my students and colleagues, so starting next week I will be on strike and not doing any work that brings any benefit to my employer - and unfortunately my academic work is decidedly within the scope of struck work. The strike will last as long as June 30th when the grad student contracts expire, but may end sooner than that, pretty unclear at this point. hope u all understand

https://www.uaw4811.org/2024-ulp-charges

sneakers-the-rat commented 2 days ago

back at work and ready to roll. got numpydantic linkml PR open https://github.com/linkml/linkml/pull/2178

and about to update nwb-linkml to reflect all the work we've done with linkml arrays. my goal is to get read and write working with HDF5 and zarr, and demonstrate a yaml-based NWB form like

my_dataset: !core.nwbfile.NWBFile
  # ... various metadata fields
  acquisition:
    probe_0_lfp: !core.ecephys.LFP
      probe_0_lfp_data: !core.ecephys.ElectricalSeries
        data:
          # ... metadata attributes
          array: # specify using a relative path and hash
            path: probe_0_lfp_data.zarr
            hash:
              value: # some long hash
              type: blake2b
          electrodes: 
            table: !reference /general/extracellular_ephys/electrodes
            array: [0, 1, 2, 3, 4, 5] # inline arrays should work the same as path references
  general:
    extracellular_ephys:
      electrodes:
        # ... and so on
  stimulus:
    presentation:
      my_stimulus_video: !core.base.TimeSeries
        data:
          array: 
            path: my_video.mp4  # videos behave the same as arrays
            hash: # ...

in some serialized form, just as an example. which is sort of like what's going on in hdmf_zarr except it's human readable, it can accept any kind of file (going to add an additional metadata field for plugins needed to read/write files), and it doesn't become dependent on hierarchical folder structure (that can still be used if it's helpful, but in general I think relying on directory structure in formats is another point of coupling between the abstract specification and the concrete implementation that seems to put a low cap on their expressiveness).

sneakers-the-rat commented 1 day ago

sneek preview of the schema models https://github.com/p2p-ld/nwb-linkml/tree/linkml-arrays/nwb_linkml/src/nwb_linkml/models/pydantic/core and the generated schema: https://github.com/p2p-ld/nwb-linkml/tree/linkml-arrays/nwb_linkml/src/nwb_linkml/schema/linkml/core

that was super easy. need to make some changes in upstream linkml and i think i can scrap pretty much all of my monkeypatching of the generator too. then it's just a matter of writing dumpers and loaders.

one of those times where you thank yourself for overengineering something before bc it was extremely simple to just swap out the translation/generation routine here. i think u all are gonna get a kick out of nwb-linkml once it's working by how simple it is