Interaction with NDArray

mikera commented 10 years ago

Came across a slightly odd bug, for some reason only seems to trigger on Travis CI builds.

It appears that core.logic is trying to turn an NDArray into a sequence, which fails when the NDArray is 0-dimensional (i.e. a scalar-sized NDArray). NDArray is sequential? so presumably core.logic is trying to unify with it?

Not quite clear to me where the defect is, seem to be two possibilities:

Is NDArray behaving badly, i.e. returning a 0-dimensional array when it should be returning a scalar?
is expresso making some incorrect assumption about how arbitrary core.matrix arrays behave?

ERROR in (test-differentiate) (ndarray.clj:1169)
expected: (= 2.0 (differentiate (quote [x x]) (ex (** x 2))))
  actual: java.lang.IllegalArgumentException: can't get slices on [0]-dimensional object
 at clojure.core.matrix.impl.ndarray$row_major_seq.invoke (ndarray.clj:1169)
    clojure.core.matrix.impl.ndarray$row_major_seq_no0d.invoke (ndarray.clj:1169)
    clojure.core.matrix.impl.ndarray.NDArray.seq (ndarray.clj:1169)
    clojure.lang.RT.seqFrom (RT.java:491)
    clojure.lang.RT.seq (RT.java:486)
    clojure.core$seq.invoke (core.clj:133)
    clojure.core.logic$eval2586$fn__2587.invoke (logic.clj:2403)
    clojure.core.logic.protocols$eval1267$fn__1268$G__1258__1275.invoke (protocols.clj:198)
    clojure.core.logic$force_ans$fn__2605.invoke (logic.clj:2432)
    clojure.core.logic.Substitutions.bind (logic.clj:410)
    clojure.core.logic$enforce_constraints$fn__2533.invoke (logic.clj:2310)
    clojure.core.logic.Substitutions.bind (logic.clj:410)
    clojure.core.logic$reifyg$fn__2545.invoke (logic.clj:2329)
    clojure.core.logic.Choice.bind (logic.clj:1070)
    clojure.core.logic$eval1834$fn__1843$_inc__1844.invoke (logic.clj:1106)
    clojure.core.logic$eval1834$fn__1835$fn__1836.invoke (logic.clj:1111)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:67)
    clojure.lang.RT.seq (RT.java:484)
    clojure.core$seq.invoke (core.clj:133)
    clojure.core$take$fn__4232.invoke (core.clj:2554)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:60)
    clojure.lang.LazySeq.first (LazySeq.java:82)
    clojure.lang.RT.first (RT.java:577)
    clojure.core$first.invoke (core.clj:55)
    numeric.expresso.rules$apply_semantic_rule.invoke (rules.clj:209)
    numeric.expresso.rules$apply_rule.invoke (rules.clj:239)
    numeric.expresso.rules$apply_rules.invoke (rules.clj:311)
    numeric.expresso.rules$apply_to_end.invoke (rules.clj:279)
    numeric.expresso.rules$eval12166$fn__12167.invoke (rules.clj:408)
    numeric.expresso.protocols$eval3348$fn__3349$G__3339__3356.invoke (protocols.clj:224)
    numeric.expresso.rules$transform_expression_STAR_.invoke (rules.clj:363)
    clojure.core$map$fn__4207.invoke (core.clj:2485)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:60)
    clojure.lang.RT.seq (RT.java:484)
    clojure.core$seq.invoke (core.clj:133)
    clojure.core$dorun.invoke (core.clj:2780)
    clojure.core$doall.invoke (core.clj:2796)
    numeric.expresso.rules$eval12166$fn__12167.invoke (rules.clj:405)
    numeric.expresso.protocols$eval3348$fn__3349$G__3339__3356.invoke (protocols.clj:224)
    numeric.expresso.rules$transform_expression_STAR_.invoke (rules.clj:363)
    numeric.expresso.rules$transform_expression.invoke (rules.clj:382)
    numeric.expresso.calculus$differentiate.invoke (calculus.clj:80)
    numeric.expresso.core$differentiate$fn__18993.invoke (core.clj:250)

mikera commented 10 years ago

Temporary fix is to drop core.matrix dependency back to 0.10.0

mschuene commented 10 years ago

I investigated this issue a little bit. The problem is the behaviour of NDArray with empty shape. PersistentVector returns just the number in this case while NDArray has these wrapper types which are problematic for me because expresso treats elements with [] as normal numbers. Also it relied up to now on the behaviour that new-array returns a zero filled vector with the given shape. As I look into the docstring now it says it will be initialized with the default value of the implementation and that is null for ndarray. I would like the behaviour of new-vector generalized here for arbitrary shape. Note that ndarray fills with 0.0 with new-vector. Maybe a fill method in core.matrix would be a good idea. I see there is already a fill! method. The other huge annoyance is that an NDArray doesn't behave like a vector: assoc doesn't work and more important count doesn't work. Switching count to ecount introduces a bug because count is only concerned with the top level. I think it is very important to support the normal clojure vector api on NDArray especially if it is the default implementation considering how fast you get ndarrays from core.matrix functions even if you never specified it.

mikera commented 10 years ago

@si14 copying you in on this because you probably have some good ideas :-)

Some points:

a new-array variant that fills with zeros is probably a good idea. Alternatively :ndarray-double should do zero filling by default
fill sounds sensible for API completeness, although assign should already do the job.
NDArray probably can implement IPersistentVector - I think it's a good idea in principle but we need to think about the zero dimension case (is that consistent with IPersistentVector??)
You should be able to use (dimension-count m 0) to get count like behaviour on an arbitrary array. I've added a slice-count function that does this for the next version

mikera commented 10 years ago

@mschuene - NDArray with empty shape is the old zero-dimensional array issue again, right?

Something with a shape of [] could currently be either a scalar value or a 0-dimensional array. Actually I've been wondering if a scalar value should have a shape of nil - would that help you?

Apart from that, it looks like core.logic is traversing the NDArray via seq, right? I'm surprised that this is returning a zero-dimensional array, it should pure return numbers when it gets down to the zero-dimensional level

mschuene commented 10 years ago

@mikera - any news on this issue? Expresso runs fine with core.matrix 0.20.0

clojure-numerics / expresso

Interaction with NDArray #3