Low level and high level API

Siu mentioned in our meeting that it would be good to implement a high level extension which uses a low level extension.

It just dawned on me that the low level extension could closely model the corresponding C structure, whereas the high level API should correspond to the exposed python API.

For example, this is how you get the shape of a fixed dimension array from the python API:

In [3]: from xnd import xnd

In [4]: xnd([1, 2, 3]).type.shape
Out[4]: (3,)

Let's look how that is implemented. The .type gives you a ndtype pyobject. Then shape will call ndtype_shape:

static PyObject *
ndtype_shape(PyObject *self, PyObject *args UNUSED)
{
    NDT_STATIC_CONTEXT(ctx);
    ndt_ndarray_t x;

    if (ndt_as_ndarray(&x, NDT(self), &ctx) < 0) {
        return seterr(&ctx);
    }

    return tuple_from_int64(x.shape, x.ndim);
}

Which calls ndt_as_ndarray on the underlying ndt_t:

int
ndt_as_ndarray(ndt_ndarray_t *a, const ndt_t *t, ndt_context_t *ctx)
{
    int n;

    assert(t->ndim <= NDT_MAX_DIM);

    if (ndt_is_abstract(t)) {
        ndt_err_format(ctx, NDT_TypeError, "type is not an ndarray");
        return -1;
    }

    if (!ndt_is_ndarray(t)) {
        ndt_err_format(ctx, NDT_TypeError, "type is not an ndarray");
        return -1;
    }

    if (t->ndim == 0) {
        a->ndim = 0;
        a->itemsize = t->datasize;
        return 0;
    }

    a->ndim = t->ndim;
    a->itemsize = t->Concrete.FixedDim.itemsize;

    for (n=0; t->ndim > 0; n++, t=t->FixedDim.type) {
        a->shape[n] = t->FixedDim.shape;
        a->strides[n] = t->Concrete.FixedDim.step * a->itemsize;
        a->steps[n] = t->Concrete.FixedDim.step;
    }

    return n;
}

Which in turn does some math on some properties of it.

Ideally, I would like to be able to write something like ndt_as_ndarray in Python. As well as call ndt_as_ndarray from Python. So could the low level API model this C structure closely?

Then the high level API, which is what most people will use so they can write code against the Python API and jit it, can be built up from this.

xnd-project / numba-xnd

Low level and high level API #1