Closed alexmojaki closed 3 years ago
Hi @alexmojaki thanks very much for the pointers. I will take a look at that at some point. If you take a look at the article, it explains my reasoning for doing my own parsing. (I am the ANTLR guy after all haha.)
Unfortunately, I do need to reexecute all elements on the line to handle all of the matrix algebra calls. I will check out your work though to see if I can learn some more goodies.
I admit that I only looked at this briefly and only saw clarify
.
How exactly does clarify identify the correct operation? Does it evaluate all subexpressions until it hits the exception again?
You may also be interested in https://github.com/alexmojaki/birdseye which is reminiscient of explain
and astviz
. It runs a modified AST (explained a bit more here) so that it doesn't have to re-evaluate anything. There's also a simpler implementation of the same concept that only outputs text here: https://github.com/alexmojaki/snoop#ppdeep-for-tracing-subexpressions.
Yep, I try all subexpressions.
Thanks. birdseye looks cool.
There are serious performance considerations so I must let python execute the code "natively" and call into C++ etc...
Yep, I try all subexpressions.
Well for starters, using executing
would mean not having to do that, you just evaluate the actual operands and no more.
So if there's an exception in a+b+c
, after you evaluate a+b
and don't get an exception, how do you evaluate a+b+c
without evaluating a+b
again? Do you replace the string a+b
with a temporary made up variable and evaluate a new string _temp+c
?
There are serious performance considerations so I must let python execute the code "natively" and call into C++ etc...
Not sure what you mean by this. birdseye/snoop.pp.deep work by essentially turning each expression x
into something like after_hook(before_hook("x"), x)
. This means that Python evaluates before_hook("x")
, x
, and after_hook("x", x)
in that order. There's inevitable overhead involved in calling those hooks to record these evaluations but no more than however you're doing it. Python does all the evaluation and C calls happen as normal.
Speaking of performance, isn't it a problem to multiply giant matrices twice? Even if there's no dangerous side effects, doesn't that take twice as long?
Basically I wanted something that did not in any way affect the interpreter or how code was executed, nor did I want to create my own interpreter hooks that managed the trees etc. I also needed to handle more than just operators. You should take a look at the article as it explains all the capabilities.
I definitely don't want to instrument the code with hooks as you have. When I talk about performance I'm talking about evaluation before it gets to my code; e.g., your code instrumentation can screw up all sorts of optimizations, debugger use, translation to cython etc.; could also slow things down just by having those Python calls in there.
Anyway, I'm happy with the way things are for the specific purpose I designed it for.
Hi! I stumbled across this library and noticed I could help. I've written a couple of libraries that are great for this stuff:
Here is a demo of how you could use it for this kind of project:
To run this you will need to
pip install executing pure_eval asttokens
.This should improve the parsing and such significantly. For example this will handle line continuations just fine. pure_eval will only evaluate simple expressions to avoid accidentally triggering side effects.
This uses the
ast
module from the standard library. Is there a reason you wrote your own parser? The best place to learn aboutast
is here: https://greentreesnakes.readthedocs.io/en/latest/I'll let you integrate it into your code yourself, but let me know if you have questions.