Closed boringcactus closed 2 years ago
I see at least currently you're using a custom tokenizer and parsimonius as a sequence matcher on top of that.. how is that approach? https://git.sr.ht/~boringcactus/crowbar-reference-compiler/tree/main/item/crowbar_reference_compiler/parser.py
I too want better tokenization than parsimonious seems capable of, and what you have there seems reasonable for your tasks, but I was hoping for a smaller solution for a smaller project.
@habnabit that approach worked fine for that project, but i had to be extra sure my regexes were getting tested in the right order
I'm using Parsimonious to implement the reference compiler for a language I'm working on, and in the absence of maximally-Unicode-expressive regular expression support (would be fixed by #162 or eventually abandoning regular expressions for something more well-scoped) I'd like to use a custom tokenizer and the TokenGrammar class. However, the
__str__
method onParseError
assumes itstext
property is a string rather than a token sequence: https://github.com/erikrose/parsimonious/blob/master/parsimonious/exceptions.py#L35I'm not quite sure how this should be fixed, but it does make debugging and error reporting more difficult.