Core.matrix integration?

[mikera]

This is a somewhat open-ended issue, I'm interested in effective integration with core.matrix and Apache spark and how best to achieve this.

Some use cases that are potentially of interest:

• extracting and/or viewing a spark RDD as a core.matrix array
• using core.matrix operations within spark
• loading core.matrix arrays into spark RDDs

[chrisbetz]

Hi,

I do not have any experiences with core.matrix so far. On a first look, I see the following issues:

• you need to distribute the implementing library to each Spark executor, which might be a cause of trouble if you rely on native implementations. Haven't tested that yet.
• serializing Protocol calls is not possible at the moment, so you need to wrap each call with a native function call (see http://dev.clojure.org/jira/browse/CLJ-1701)

If you got some concrete issues I might help you with just drop me a note.

Cheers,

Chris

[chrisbetz]

Closing this ... won't provide an integration out of the box within sparkling.