Consider native serializer support for numpy.ndarray

[gsmecher]

Things to check first

• [X] I have searched the existing issues and didn't find my feature already requested there

Feature description

Currently, the easiest way to serialize numpy.ndarrays using cbor2 is something like (neglecting error checks)

import numpy as np
import cbor2

x = np.ones(10)
y = cbor2.dumps(x, default=lambda x, y: x.encode(y.tolist()))

This requires numpy to traverse the array and convert it to a Python list, which is then handed off to cbor2 for another traversal - there are several traversals and transient allocations involved.

Because both Numpy and CBOR have clean C APIs, would you consider a direct conversion implemented in the C extension module? It's worth noting that the orjson JSON library does this already.

Use case

Low-overhead serialization of numpy arrays.

[agronholm]

I'm open to the idea if this can be done cleanly. What should ndarrays serialize to, in CBOR terms?

[gsmecher]

Oops, I see this is already discussed in #59.

It looks like there are some options:

1. Homogeneous typed arrays per RFC 8746
2. Classic CBOR arrays with individual type tags
3. Some numpy-specific type tag

The combination of (1) and (2) seems ideal, with (1) as a fastpath and (2) as a fallback. I'm optimistic Python ndarrays carry enough type metadata to decide between them without traversing the array.

(3) seems easy to rule out, and I'm only including it to say so out loud.

[agronholm]

Option 1 sounds like the best for encoding, but decoding may be an issue, particularly when numpy isn't present.

[gsmecher]

Note we've got an out-of-tree implementation here:

https://github.com/gsmecher/tuberd/blob/master/tuber/codecs.py

It's BSD 3-clause - if there's any ambiguity about whether you can borrow code I'm happy to chase down the contributor and ask about relicensing. I'd be happy to see this code end up in your project, and we'll eventually be able to remove it from ours.

[agronholm]

Nice to hear, but right now I don't have any bandwidth for this, as my attention is currently focused on AnyIO, APScheduler and Typeguard.

[gsmecher]

Bandwidth is a terribly scarce resource. :) Thanks for all of your work.