Variable length data types

Resolves #21.

This is a substantial change that adds support for variable length data types to zarrs. There were some breaking changes necessary to support this:

Array store/retrieve "bytes" methods now take/return ArrayBytes which can represent fixed or variable length bytes, rather than just a slice-like
Array store/retrieve "elements" variants use new Element[Owned] traits, with better validation
Encoded bytes are aliased to RawBytes
Codec traits have had some changes to accommodate the distinction between ArrayBytes and RawBytes

Data types

String (utf-8)
Binary

Codecs

`vlen`

{
  "name": "vlen",
  "configuration": {
    "data_codecs": [{"name": "bytes"},{"name": "blosc","configuration": {"cname": "zstd", "clevel":5,"shuffle": "bitshuffle", "typesize":1,"blocksize":0}}],
    "index_codecs": [{"name": "bytes","configuration": { "endian": "little" }},{"name": "blosc","configuration":{"cname": "zstd", "clevel":5,"shuffle": "shuffle", "typesize":4,"blocksize":0}}],
    "index_data_type": "uint32"
  }
}

Based on https://github.com/zarr-developers/zeps/pull/47#issuecomment-1710505141.

Structure:

a little-endian uint64 representing the size in bytes of the encoded index,
index data structured using the Apache arrow variable-size binary layout with the validity bitmap elided (https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) and encoded with index_codecs,
contiguous element bytes encoded with data_codecs.

The encoded index size is necessary to support index compression and partial decoding. If this were not available, the index could not used a bytes-to-bytes compression codec. A bytes-to-bytes compression codec could follow vlen, but then "data" is potentially running through a compression codec twice.

`vlen_v2`

{
  "name": "vlen_v2"
}

This matches Zarr V2 style interleaved encoding, which is implemented by numcodecs vlen-utf8, vlen-bytes, and vlen-array. These are all essentially the same codec, with data type-dependent behaviour. It makes sense to standardise a single codec for Zarr V3 to support Zarr V2 vlen-utf8/bytes/array encoded data without reencoding chunks.

Encoding Efficiency (32-bit index)

Sum of chunk sizes (in bytes) on "city" column of https://github.com/zarr-developers/zarr-python/pull/2036#issuecomment-2227440951.

https://github.com/LDeakin/zarrs/blob/variable_length_data_types/tests/cities.rs.

encoding	compression	size
vlen_v2		642196
vlen_v2	zstd 5	362626
vlen		642580
vlen	zstd 5	346950

Files	Patch %	Lines
src/array/element.rs	65.41%	46 Missing :warning:
...rray_to_bytes/sharding/sharding_partial_decoder.rs	88.28%	39 Missing :warning:
src/array/array_sync_sharded_readable_ext.rs	63.95%	31 Missing :warning:
...en_interleaved/vlen_interleaved_partial_decoder.rs	53.03%	31 Missing :warning:
src/array/codec/array_to_bytes/vlen.rs	71.13%	28 Missing :warning:
src/array/array_bytes.rs	93.63%	25 Missing :warning:
...o_bytes/vlen_interleaved/vlen_interleaved_codec.rs	78.72%	20 Missing :warning:
src/array/array_representation.rs	56.75%	16 Missing :warning:
src/array/codec/array_to_bytes/vlen_interleaved.rs	68.00%	16 Missing :warning:
src/array/array_async_readable_writable.rs	68.08%	15 Missing :warning:
... and 27 more

LDeakin / zarrs