Javascript expression ast/parser/builder

I threw together a little AST class built on top of interned immutable tuples, and wired it up to an acorn parser that ought to be able to accommodate everything in pygrim/formulas -- and anything you could generate with fungrim's Expr class. Here's a sneak preview. The parser should also handle all the same operators overloaded by fungrim, and in the same way (and if not, it should be easy enough to add/fix, since the subset of python used by pygrim syntax happens to also be a proper subset of javascript).

I still need to toss in wire up some latex printing, among other things, but this be a pretty decent start. I also have a little node builder helper I've been hacking up which uses a proxy handler, which would give ~ the same semantics as pygrim's Expr builder (sans operator-overloading, of course. Although we do get === equality of expressions by way of the interned tuples we use for the underlying term boxes. And when I wire in evaluation, we can use the Symbol.toPrimitive hook to get == equality of numerically evaluated expressions.

I have another variant of this that mirrors the pygrim Expr API much more closely, including surfacing a factory function facade for each Expr node, but this was a little less pleasant to work with since it required all the same inject_builtin/inject_vars calls as fungrim to be usable, and they either had to be dumped into global scope, or into a context object which would have to prefix every symbol reference. Using a proxy handler is a lot more flexible.

Once I get some regex printing in it should be easy enough to refactor into a lib you could pull into fungrim for rendering from within node. Let me know if this seems useful (and/or wildly off-the-mark).

Seems like a good start.

I think some kind of client-side numerical evaluation widget directly interfacing Arb (without the need to go through symbolic expressions) would be a more useful near-term addition, but feel free to work on this :)

A feature that is missing for the backend right now is to export expressions to different programming languages, which may include operator overloading. This just has to be done a bit carefully because you want to avoid constant folding, e.g. Pow(2,2) -> 4, and worse Div(1,3) -> 0.3333333333333333. (We could then, among other things, just export the database to whatever format is easiest to read for JS.)

I think some kind of client-side numerical evaluation widget directly interfacing Arb (without the need to go through symbolic expressions)

Indeed -- that's what I'd started working toward with that little arbuckle lib, but in order to do support generic evaluation, I wanted to work through the semantics for a consistent data structure for the AST though. Specifically, the immutability/value-type semantics and serialization, in order to simplify js/wasm object lifecycle concerns, and still preserve the ability to leverage arb's iterative refinement for numerical evaluation. Also simplifies interfacing with an arb wasm lib as a worker -- which is pretty important in order to do anything particularly interesting, like rendering a complex plot without janking up the ui.

(edit: a semi-related reason, I also have a variety of special functions implemented, e.g. over specific structures, either directly in js and wasm (for instance, the whole family of arithmetic functions map beautifully over prime factorization "bags"). This would finally gives me a place to hang all these different implementations in a place where they could be readily accessed, verified, compared, improved... so long as I can get the evaluation/compilation right.)

A feature that is missing for the backend right now is to export expressions to different programming languages, which may include operator overloading. This just has to be done a bit carefully because you want to avoid constant folding, e.g. Pow(2,2) -> 4, and worse Div(1,3) -> 0.3333333333333333. (We could then, among other things, just export the database to whatever format is easiest to read for JS.)

Heh -- I know this pain all too well. Which is why this is fully supported, and ought match python semantics and operator precedences. I have to clean this up, and extract out the various pieces into npm libs, but search the page for the pygrim template literal, which will parse any pygrim input expression/formula into an immutable tuple AST structure...

pygrim`1/3` === pygrim`Div(1,3)` // true
pygrim`1/3` === Term.of(Term.ident('Div'), 1, 3) // true
pygrim`1/3`.repr === "Div(1,3)" // true
pygrim`2**2`.repr === pygrim`Pow(2,2)`.repr === "Pow(2,2)" // true

The basic AST knows nothing of these fungrim symbols or operators -- ultimately it can be completely agnostic to any symbol (for now, it only knows about integer/text/symbol atoms). It's the pygrim_parse method that's doing this rewriting from, say, the binary ** operator to Pow(2,2). I've only added the operators overloaded by pygrim, and I'm pretty sure precedence ought to match between js and python (and if not, I can fix up in a rewriter pass or a preprocessor). I was even able to add support for python-like tuples (with length > 1) by trapping and transforming SequenceExpression form.

On a related note, check out the little grimtex template literal -- it's able to render a large proportion of fungrim identically to what's in pygrim's latex.py. There's still a fair bit missing, and still need to integrate into AST, but should give you a rough idea of what I'm after -- I want the ability to write equations easily and elegantly (e.g. in a text area) and get beautiful output and evaluate them on the fly... And pass into any one of a number of coherent plot commands that understand this, etc. And I want to be able to do this from within markdown, or a node repl, or anywhere else (e.g. a pdf or svg, a react or react-native app, by transforming boxes to react-primitives nodes, ... yes, I'm greedy -- I want it all).

It's still a bit tangled, but I now have a pretty clear idea of what the fungrim/pygrim-specific bits are (generic immutable AST, and separately, the fungrim-specific parser, symbol def/unification, and associated semantics, e.g. evaluation and printing), and will split off into a standalone node package fungrim could depend on directly.

Cool!

I apologize for the expression language still being changed rapidly which makes it a moving target for what you're doing. I think we will eventually need a test suite to make sure the implementations stay in sync; the best way to do it would be to just export all the expression -> latex conversions for Fungrim produced by pygrim and check that jsgrim generates exactly the same output for the supported symbols.

On that note, I've started a writeup to document the expression language in more detail:

https://github.com/fredrik-johansson/fungrim/blob/master/grim.txt

This is still an incomplete draft, and there are some tentative ideas (like the vararg generators) that I haven't implemented on the Python side yet. Let me know if you have opinions on anything related to the syntax or semantics -- it's still easy to make changes!

I apologize for the expression language still being changed rapidly which makes it a moving target for what you're doing.

Actually, it shouldn't be a problem at all. Feel free to change at a whim. I'm trying to be fairly generic with the base implementations of these different mathematical objects, trying to find the right primitives so that I can stitch together a fungrim-specific "basket" of "tag heads", where the semantics of each component are just straightforward rewrites into some smallish set of fundamental set of system primitives (more here¹...)

I think we will eventually need a test suite to make sure the implementations stay in sync; the best way to do it would be to just export all the expression -> latex conversions for Fungrim produced by pygrim and check that jsgrim generates exactly the same output for the supported symbols.

That's be great, yeah. Though I can actually do you one better -- since I ought to be able to run pygrim within js via pyodine. So given any arbitrary set of expressions, I could verify the js output against pygrim's. In this way I can compare the latex string output, but also other products, like repr "FullForm" serialization, or even the results of numerical evaluation (once I get that in).

On that note, I've started a writeup to document the expression language in more detail: ...

Thanks, this is great stuff! I've been pretty tied up with work stuff lately, but will follow up with any questions/comments as I keep digging in. In the meantime, I'll throw together an example of running pygrim within js, and comparing latex string output, given a list of pygrim expressions.

[1]: For example, imagine a small set of presentation-specific primitives akin to mm's *Box elements, where most of the stringification logic for tex printing might live (as well as html, mathml, or any other serialization). Then it's just a matter of mirroring pygrim's latex semantics, but in a way that lays bare the actual presentational intent. So it ought to be relatively easy to align with however you opt to print latex in pygrim.

The same goes for evaluation, where fungrim elements are inherently HoldAllComplete (again, borrowing mm's language as it's pretty well established at this point). For numerical evaluation, you might imagine all these "NumericFunction" elements having any number of handler functions registered to support specific input expression patterns, where these patterns could include testing against specified numeric-evaluation options. In a fungrim-specific configuration, the arb-specific evaluator would be selected for automatically.

I know this is all a bit hand-wavy -- just trying to sketch out the overall direction I'd love to go, and how this kind of approach could stay completely out of fungrim's way, while still potentially adding value to fungrim itself.

fredrik-johansson / fungrim

Javascript expression ast/parser/builder #18