Order of `tokens :-` specification in .x file matters

[christabelneo]

It was not mentioned in the documentation but it seems that the order in which the tokens are defined in the tokens :- section matters? Putting some rules above others resulted in lexical errors from the generated parser, which disappeared when I changed the order of the rules. Is this due to the fact that some of my rules use regexps which contain characters which are also defined as tokens?

[simonmar]

See https://www.haskell.org/alex/doc/html/alex-files.html, in particular "When the input stream matches more than one rule, the rule which matches the longest prefix of the input stream wins. If there are still several rules which match an equal number of characters, then the rule which appears earliest in the file wins."

Does that help?

[christabelneo]

I completely missed that line, thanks! However I'm still getting some strange behaviour, when the shorter rule is above it does not work, it only works when the longer rule is above.

For clearer reference, the shorter rule is "<" { \p s -> TokenLT p } and the longer rule is $alpha $idchar* \<$alpha $idchar*(\:\[$alpha $idchar*\])? (\,$alpha $idchar*\:\[$alpha $idchar*\])? \> { \p s -> TokenIdent p s }

[andreasabel]

@christabelneo, please provide a complete MWE in the style of the tests in the tests/ subdirectory, e.g. https://github.com/simonmar/alex/blob/master/tests/basic_typeclass.x . (These examples contain both a lexer definition and a sample input that illustrates the problem/functionality.)

[andreasabel]

@Ericson2314 : Please give me some basic rights on this repo so that I can assign myself to issues, label issues, close issues etc.

[Ericson2314]

@andreasabel I'm afraid I don't have permissions to do that, but that sounds like a good plan. @simonmar can you?

[andreasabel]

This user question has been answered.