toddATavail
commented
4 years ago I wrote this on April 1, but it bounced, so here it is directly via the web form.
We have a hard and fast goal for errors with the project — an internal error / assertion failure should never be produced. Put another way, except in the cases that a user adds new code at the implementation level (Java/Kotlin) or invokes arbitrary pojo operations in an unsound way, the user should never encounter a hard crash from the compiler or the virtual machine, no matter what weird thing they tried to do in their Avail program.
Lexers are an advanced feature, but nevertheless, it is not appropriate for them to just crash the system when misused. I experienced a great deal of pain building the library lexers, because the system is not very graceful when you make a mistake.
This is definitely an area that needs extensive hardening and improvement. Because the number of users building lexers is low, however, and the capacity of the team is limited, this work unfortunately can't receive a great deal of priority right now. We should definitely revisit once the ongoing Kotlinization effort is complete.
I'll leave it to Mark to weigh in if he has other opinions.
Thank you for making such a thorough report!
I'll start by acknowledging that custom lexers obviously live much more towards the "power user side" of the current library capabilities. It's likely an open question whether making them more user-friendly is worth any current effort or not, and whether any misbehavior should attempt to be diagnosed or simply left in undefined behavior territory.
It's not hard to write a bad lexer, but here's a particularly ignorant one:
In normal operation (without assertions) attempting to build a module with access to this lexer doesn't necessarily cause issues on its own, but it mucks up the error reporting, causing the compiler diagnostics to crash and surfacing a rather opaque
ArrayIndexOutOfBoundsExceptionwithin an Avail descriptor implementation, from attempting to copy a slice of source text beyond its length.❌
ArrayIndexOutOfBoundsExceptioncrashing workbenchWith assertions, the lexer tends to fail during its operation. In my case it tended to trip up
LexingStateon an assertion that its position was not beyond the end of the source (below).❌
AssertionErrorin lexing stateHowever I suspect this is hardly the only assertion a malicious lexer could uncover; I have not experimented much in detail on that front.
Again, since this is the deep end, the impact of these problems is very low. If you're playing with lexers you probably know you broke it. But there are a few philosophical discussions here that I think might shape development goals: is there any valid recovery when the system realizes a lexer isn't doing its job? And more generally, what is the level of responsibility that, say, the workbench takes in trying to prevent arbitrarily bad code from breaking the host process?