Patch for allowing custom LexToken class

[GoogleCodeExporter]

I needed to have a filename attached to each token.
One hacky way is to replace the LexToken class with my own version after import 
(ie "lex.LexToken = MyTokenClass").
A better way would be to have the lexer construct tokens from a variable 
holding the class rather than a hardcoded name.
The attached patch realizes this.

My token class can copy the file name as part of the constructor.

It might be better to also have a function call that delivers the token data to 
the new token (eg the token constructor??).
That would solve the problem of deciding when it is safe to read eg 
token.lineno. I don't know how that would affect lexer speed though, so I left 
this change out.

Original issue reported on code.google.com by Alberth2...@gmail.com on 9 Jun 2010 at 6:59

Attachments:

• custom_tokenclass.patch

[GoogleCodeExporter]

Update: A choice is made to 'abuse' the current system, and have 'lineno' point 
to a Position object instead of an integer. In that way, lextokens get the 
right object automatically with zero effort, as long as position updates are 
done by creating a new position object.

It is not the prettiest solution, but it seems like an effective solution for 
now.

Original comment by Alberth2...@gmail.com on 17 Jun 2010 at 7:04

[GoogleCodeExporter]

PLY seems to have unnecessary limitations to the line number:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "nml/tokens.py", line 230, in dump_tokens
    fp.write(str(tok) + '\n')
  File "/usr/lib/python2.6/site-packages/ply/lex.py", line 71, in __str__
    return "LexToken(%s,%r,%d,%d)" % (self.type,self.value,self.lineno,self.lexpos)
TypeError: %d format: a number is required, not LinePosition

Due to the hard-coded %d for the lineno, it is impossible to print a token with 
a Position object as line number.

Suggested fix: Change %d to %s ("%s" % 123 outputs the number, so that solution 
is backwards compatible).

Original comment by Alberth2...@gmail.com on 15 Sep 2010 at 8:18