Python's RE module causes the wrong Token to come out of the Lexifier

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Create Tokens, where one is a broadly generic version of the second
2. Declare them both as functions (probably not necessary, but it
guarantees priority) with the specific case listed first.
3. Try parsing a file with this lexical structure, to see it in action, use
the attached files please.

What is the expected output? What do you see instead?
When reaching the line that says #begin environment support, I expect a
K_BEGIN_ENV_SUPPORT token.  Instead the Parser gets a K_ANYTHING token, and
uses the wrong rule.

What version of the product are you using? On what operating system?
Version 2.3, downloaded straight from the website. Running Fedora 7
(tragically), using eclipse for development and debugging (version 3.2 with
thelatest release of pydev 1.3.4)  I made a small change, so that I can see
all the groups from the Match object, in lex.  I'm including my copy of
lex.py and yacc.py

Please provide any additional information below.
As far as I can tell, lex seems to compile all the regexs into one massive
regex, with parameters.  I plan on making a patch where it will instead
iterate through the regexs until one matches.  This will probably be
slower.  Please let me know if this is desired, or if there will be another
solution in the works.

Original issue reported on code.google.com by loupgaro...@gmail.com on 25 Jun 2007 at 3:00

Attachments:

• AST.py
• KefirLexer.py
• KefirParser.py
• ParselToungue.py
• lex.py
• yacc.py

[GoogleCodeExporter]

I think i might have fixed it with a preliminary patch to lex.py, basically it
iterates  through all possible regexs one by one.  I'm getting other errors in 
my
code now, but I'm gonna debug it later.  I haven't thouroughly tested it, so 
please
enlighten me what the standard method for testing ply is :)

Original comment by loupgaro...@gmail.com on 25 Jun 2007 at 7:03

Attachments:

• lex.py

[GoogleCodeExporter]

Can't reproduce. Please resubmit if bug persists in PLY-3.2 or newer.

Original comment by dbeaz...@gmail.com on 27 Aug 2009 at 1:18

• Changed state: WontFix