First pass at native NDArray support.

[cmungall]

See

• https://github.com/linkml/linkml/issues/895
• https://github.com/linkml/linkml-arrays/issues/5

This introduces first-class array support into LinkML.

A minimal example would be:

    attributes:
      temperature_matrix:
        range: float
        array_info:
          exact_dimensions: 3

The native serialization of this in json/yaml will be a LoLoL. Using linkml-xarrays it will be possible to serialize using hdf5/zarr/etc.

The corresponding nptyping type would be NDArray[Shape["*, *, *"], Float].

(note: modelers will want the ability to use ctypes but this is orthogonal)

Note that this does not force any metadata on the array; we are deferring on the datamodel for what is equivalent to xarray DataArrays, these will be supported via implements for now and first-class incorporation in a future version. This will allow binding between axes are other LinkML arrays.

Minimal metadata can be introduced via naming the axes

    attributes:
      temperature_matrix:
        range: float
        array_info:
          exact_dimensions: 3
          dimensions:
            x:
            y:
            z:

The corresponding nptyping type would be NDArray[Shape["* x, * y, * z"], Float].

The shape can be further constrained; imagine an RGB matrix with coords x, y, and a length 3 r/g/b:

    attributes:
      rgb:
        range: float
        array_info:
          exact_dimensions: 3
          dimensions:
            x:
            y:
            rgb:
              exact_cardinality: 3
              description: r, g, b values
              annotations:
                names: "[red, green, blue]"

corresponds to NDArray[Shape["* x, * y, 3 rgb"]

For now if you do want to bind dimensions to additional metadata this can be done via annotations:

classes:

  TemperatureDataset:
    tree_root: true
    annotations:
      array_data_mapping:
        data: temperatures_in_K
        dims: [x, y, t]
        coords:
          latitude_in_deg: x
          longitude_in_deg: y
          time_in_d: t
    attributes:
      name:
        identifier: true
        range: string
      latitude_in_deg:
        required: true
        range: float
        multivalued: true
        unit:
          ucum_code: deg
        array_info:
          exact_dimensions: 1
      longitude_in_deg:
        required: true
        range: float
        multivalued: true
        unit:
          ucum_code: deg
        array_info:
          exact_dimensions: 1
      time_in_d:
        range: float
        multivalued: true
        implements:
          - linkml:elements
        required: true
        unit:
          ucum_code: d
        array_info:
          exact_dimensions: 1
      temperatures_in_K:
        range: float
        multivalued: true
        required: true
        unit:
          ucum_code: K
        array_info:
          exact_dimensions: 3

[rly]

Notes:

• It is possible to have scalar arrays in many array formats (e.g., NumPy, HDF5, Zarr) but not all (e.g., Matlab). They seem primarily used to allow API functions to work seamlessly with both arrays and scalars and as a box to pass scalars to a function by reference. Matlab gets around this by not having special dtypes for scalars and not allowing scalar arrays - they are just 1x1 matrices. I do not know that we need to support scalar arrays. To be conservative, perhaps it would be best to leave out support until there is a need. Refs:
  • Interesting comment on why scalar arrays exist in numpy
  • Description of scalar dataspaces in HDF5
  • Use of 0-D arrays as a data structure in Julia
  • Use of 0-D arrays as a data structure in Lisp
  • Or put attributes on an HDF5 dataset that is a single value (but you can still do that with a dataset of shape (1, ))
• If not specified, minimum_number_dimensions is 1, maximum_number_dimensions is Infinity.
• It is possible to specify minimum_number_dimensions without maximum_number_dimensions. We have a use case for a min without a max, but I cannot think of one for a max without a min.
• I am not sure there is a use case for minimum_cardinality and maximum_cardinality. Maybe we can omit them and add them later if there is a need? I am concerned about adding features to the metamodel that are useless but need to be supported for a long time to avoid breaking backwards compatibility.

Some useful validation checks:

1. The number of items in dimensions_info must not exceed maximum_number_of_dimensions
2. Each dimension_index must not exceed maximum_number_of_dimensions - 1
3. Each dimension_index must be unique across dimensions_info items
4. If array_info exists, then multivalued must be True (if we allow scalar arrays, then this is no longer always true)
5. minimum_number_dimensions <= maximum_number_dimensions
6. Cannot have both exact_number_dimensions and minimum_number_dimensions or both exact_dimensions and maximum_number_dimensions
7. minimum_number_dimensions, exact_number_dimensions, maximum_number_dimensions all > 0
8. minimum_cardinality <= maximum_cardinality
9. Cannot have both exact_cardinality and minimum_cardinality or both exact_cardinality and maximum_cardinality
10. minimum_cardinality, exact_cardinality, maximum_cardinality all > 0

[sneakers-the-rat]

for posterity, archived version of matrix of tradeoffs to be added to consolidated docs later, along with @rly 's notes and examples :)

live: https://wiki.jon-e.net/LinkML_Arrays archive: https://web.archive.org/web/20240207234701/https://wiki.jon-e.net/LinkML_Arrays