Closed iand675 closed 4 years ago
Is this still an issue with the latest release?
Yes.
$ alex Issue71.x
alex: Prelude.head: empty list
$ alex --version
Alex version 3.2.4, (c) 2003 Chris Dornan and Simon Marlow
(2003
should be either dropped or updated.)
The following call to head
throws the error:
https://github.com/simonmar/alex/blob/f2c343b9b20ff9508f482e87528ef486c55b9f86/src/DFAMin.hs#L58
The remaining problem is that if the lexer definition contains nullable tokens and the Alex gets stuck (on EOF or no just no possible DFA-transition), it outputs an infinite sequence of the nullable tokens. This happens because alex_scan_tkn
gladly returns accepts of length 0.
The problem can be fixed by checking for the scanned length here:
https://github.com/simonmar/alex/blob/0198f7387a8f31fd8a7ff3efe453edbb00cba4fb/templates/GenericTemplate.hs#L155
and if the length is still 0, output AlexNone
instead of accept or skip.
Why should a lexer not return tokens of length 0? Well, because the number of these token sequences is not determined, it could output any number of such tokens at any point. Thus, outputting 0 of such tokens only is a sensible (and the only sensible) choice.
I'd say this issue is fixed now and can be closed.
It looks like the zero-or-more matching functionality in Alex 3.1.4 has an error. In the following minimal example, alex emits the following error:
alex: Prelude.head: empty list
.In the event that I change line 7 from
@whitespaces = $whitespace*
to@whitespaces = $whitespace+
, the error goes away.