search

gilch / hebigo

蛇語(HEH-bee-go): An indentation-based skin for Hissp.
https://github.com/gilch/hissp
Mozilla Public License 2.0
24 stars 3 forks source link

Python 3.10 broke the lexer #61

Closed gilch closed 2 years ago

gilch commented 2 years ago

Hebigo's lexer relies on Python to determine when a bracketed expression is complete. It reads up to the first matching ending bracket, and asks Python to parse it. Brackets can be nested and escaped in things like string literals, meaning the first one encountered might not be the right one, so on certain SyntaxErrors it reads up to the next matching bracket and tries again. Other SyntaxErrors are unrecoverable and should propagate out. Unfortunately, the only way to distinguish these cases is by the error message, and those changed in 3.10.

I could include the new messages, but this seems just as brittle, and I feel like I could easily miss one. Another option would be to assume all SyntaxErrors are recoverable. This is probably good enough in the REPL, but it will result in poorer feedback in case of unrecoverable errors, as the kernel will simply keep asking for more lines, rather than pointing at the problem. In a .hebi file, an unrecoverable error would result in the remainder of the file lexing as one bracketed expression.

gilch commented 2 years ago

That pretty much fixed it. The REPL is maybe not as nice now. If you make an unrecoverable SyntaxError, you'll certainly notice something is up, but you'll have to find it yourself now. This should not be too difficult when forms are small, but if you paste in a big class or something, the REPL is not going to help you as much. I am not sure what to do about this yet.

Maybe if there's a way to provide feedback before accepting the form, any SyntaxErrors encountered could simply be displayed? You might get a few recoverable ones appearing while typing, but if it's not interrupting you, this is pretty harmless.

Another option would be to forget about automatically detecting the end of a form, and require an explicit Ctrl-Enter or something. This doesn't feel as nice, but it's pretty much how it works in a notebook.

gilch commented 2 years ago

Looks like Esc Enter will force-evaluate a cell in Jupyter Console. This may be good enough to locate the problem in that big class.

gilch commented 2 years ago

Also Alt+Enter. Certain terminal emulators will accept only one or the other if one of them is already bound to something else.

gilch commented 2 years ago

Python must have a way already to determine if Python input is invalid or just incomplete for its own REPL. Given code.interact, it's probably user-accessible too. I should just find that and delegate.