Improve usability of arrays/vectors

[stuhlmueller]

Some ideas from #494:

• Have clear documentation on when to use which
• Show helpful error messages when one is mistaken for the other
• Make the interface similar enough that most array functions can polymorphically apply to tensors
• Once we have ES2015 support everywhere we could implement property lookups on vectors using a Proxy

Example issue:

var xs = Vector([0, 1, 2]);
xs[1];  // => undefined
xs[1] > 0.5;  // => false

[null-a]

There should be official, documented, ways to convert arrays <=> tensors.

[null-a]

It's not currently possible to take a bunch of (lifted) scalars and turn them into a tensor. e.g. Vector expects an array of unlifted numbers.

[null-a]

In the webppl meeting (16 Aug) @ngoodman suggested that we arrange to have the arithmetic operators do the right thing for tensors. This will have to happen at run-time (cost?), and will involve rewriting operators to new functions that do the type checks and dispatch as appropriate. The new version of sweet.js doesn't yet allow custom operators to be defined, so I imagine */+- will operate element-wise and we'll still have functions for things like dot, transpose, etc.

[ngoodman]

couldn't we overload * to be matrix multiply (aka dot)?

and note that we are doing this runtime overloading anyhow for AD...

[null-a]

We also talked about using tensors throughout the system, whenever we need to represent a bag of numbers -- this uniformity will make it easier to predict what type a function/distribution constructor expects.

[null-a]

couldn't we overload * to be matrix multiply (aka dot)?

Yep, that's the other option of course! Having */+- all be element-wise seems more consistent to me, but I'm happy to implement it whichever way people think is best.

[dritchie]

couldn't we overload * to be matrix multiply (aka dot)?

In most linear algebra libraries I've seen, the infix operators do element-wise operations, and things like dot products and matrix multiplication are done via standard method/function calls. One possible compromise would be to have * do different things depending on the dimensionality of its arguments (element-wise if they have the same shape, inner product/mat mul if they have the right sizes for that, or an error otherwise). That could get confusing, though.

[ngoodman]

hmm... good old matlab just has different ops: * is matmul and .* is element-wise....

[null-a]

hmm... good old matlab just has different ops: * is matmul and .* is element-wise....

Yeah, that's what I had in mind when I mentioned that the new version of sweet.js doesn't yet have custom operators. If we wanted to end up with something like the matlab approach, than maybe we'd hold off on infix operators until sweet.js either does support this or they decide not to.

The option of implementing .* and friends using the version of sweet.js we're currently stuck on doesn't seem like a good idea.

Also, as @dritchie pointed out, by using custom operators we'd break syntax highlighting everywhere, which has some negative utility.

[null-a]

Also, reconsider whether we want to represent vectors as rank 2 tensors. (@dritchie has thoughts on this.)

[longouyang]

Could dirichlet et al be polymorphic, accepting both vectors and arrays? The idea here is that you wouldn't have to worry about vectors when prototyping a model (but if you really want the performance benefit, you can pay the syntactic cost of switching)

[ngoodman]

thought i'd revive this thread, now that we've removed sweet.js dependence.

what do folks think about overloading *, +, etc, and adding .* as part of our AD transform pass?

[null-a]

I think overloading *, + etc. will be useful, and I consider it to be one of the things on my todo list.

.* will be problematic I think, since it presumably won't be handled by the JS parser we use?

[ngoodman]

.* will be problematic I think, since it presumably won't be handled by the JS parser we use?

oh -- if we're doing our transform after parsing then yes... we'd have to do a preprocessing step to convert x .* y into matmul(x,y) or such...

[longouyang]

Some questions:

Would overloading + preserve its use for string concatenation?

For matmul versus element-wise, maybe we could use one of the existing but obscure operators like ^? Also, I think it'd be an easier change if we kept * as element-wise and introduced a new notation for matrix (rather than changing the existing meaning of * to something new)

[null-a]

I like*,+, etc. for element-wise too. I also think dot or whatever for matrix-matrix is tolerable. (Does it even come up very often?) Reusing an existing operator would be convenient but I'm not sure it's worth the confusion?

As well as having to write our own parser, another snag (pointed out by @dritchie iirc) with using .* is that it breaks JS syntax highlighting in text editors.

Would overloading + preserve its use for string concatenation?

Yes.

[null-a]

I made a first attempt at overloading arithmetic ops and have something working. I'm interested in how much overhead this introduces, so I ran a quick test using examples/linearRegression.wppl to get an initial feel. (There's some arithmetic in this program, but more in the score function. We could write the scorer in such a way that we avoid this overhead in the final thing, of course.)

It looks like my current implementation increases the run time of this program by about 20%. One way we might be able to improve on this is by combining some of the checks I'm doing before dispatching to ad.scalar or ad.tensor with the checks within ad.tensor.add etc., For example, it might be that I'm checking for (and 'unwrapping') ad graph nodes more than once at present.

In programs that use lots of large tensors, it seems reasonable to expect the overhead to be less significant than this.