core.matrix support

[mikera]

Hello,

Mandoline looks like a great library for managing multi-dimensional array data!

Would it be possible to extend mandoline to provide core.matrix support, so that a mandoline data set can be used via the core.matrix API?

Advantages:

• core.matrix already provides a fairly comprehensive generic API for accessing and manipulating multi-dimensional array data
• It would improve interop with other Clojure libraries that use core.matrix (e.g. Incanter)
• It would help us to develop a compatible ecosystem of libraries for data science in Clojure. Note that core.matrix should at some point become "official" Clojure contrib (has been approved by Rich)

See: https://github.com/mikera/core.matrix

I'm the maintainer of core.matrix, happy to make improvements as necessary to improve interoperability with libraries like mandoline. Also happy to give input / guidance on how to make this work efficiently.

Mike.

[monodeldiablo]

Hey Mike!

UCAR's Array was chosen more out of convenience than necessity, since our primary input data format is NetCDF. Using the same library (and thus, in-memory format) as our ingest processes means we avoid unnecessary copy overhead (most of the time).

Nevertheless, interoperability is the name of the game. And your help is much appreciated.

How deep into Mandoline do you see core.matrix support extending? Do you envision a view on top of Mandoline's existing Slab type, or something deeper?

[mikera]

That sounds fine: core.matrix is designed to enable use of different underlying array types.

It's probably most convenient just to extend the core.matrix protocols either to the UCAR array type and/or the Slab type - that way you would get full core.matrix API support essentially for free. This is the approach that Clatrix (https://github.com/tel/clatrix) takes, for example.

Most of the protocols are optional, you only need to implement a small number to get the API working - the rest are available for improving performance and can be added later.

P.S. As an aside you'll probably find copy overhead is trivial compared to the IO cost of the query. The only time I've found copy overhead to matter is in extremely CPU-bound numerical code.