toward storage of arbitrary data types

[bogovicj]

@minnerbe helped with this issue.

N5 core does not support non-native and non-numeric types well (i.e. all DataBlock implementations are native + numeric). E.g. the API can not currently implement / wrap HDF5's string io (see https://github.com/saalfeldlab/n5-hdf5/issues/22)

related

For string support @minnerbe implemented https://github.com/minnerbe/n5/tree/feature/vlstring-io

zarr

Zarr has support for writing object arrays, and does so via codecs in numcodecs. Existing options are JSON, MsgPack, Pickle

proposal

1) Include a generic Object DataBlock in n5

• consider a special case String DataBlock? 2) Add a codec interface
• could be similar to the existing Compression n5 interface
• idea similar to numcodecs
• For Zarr interop, we should add JSON and MsgPack encoders
  • MsgPack may a good default for good zarr interop (see msgpack-java)
  • JSON is easy to implement
• protobuf is another option to consider
• Java's object serialization would be easy to add
  • but we probably shouldn't use it
• Pickle - let's ignore

[minnerbe]

Thanks for bringing this up! I would add to point 1 of the proposal that this seems quite similar to the Object data type that is already implemented. However, Objects

• rely on the native serialization of Java and thus cannot use a custom serialization format;
• have a fixed data-type meta-data tag ("object"), which may not comply with external standards (e.g., "String(-1)" for variable length string arrays in HDF5).

It's probably worth considering to merge these two concepts and provide a codec that just uses native Java serialization.

[bogovicj]

see also https://github.com/saalfeldlab/n5/issues/87