Have "exact_type" attribute for tokens

[hydrargyrum]

With tokenize module:

>>> next(tokenize.generate_tokens(iter(['()\n']).__next__)).type == token.OP
True
>>> next(tokenize.generate_tokens(iter(['()\n']).__next__)).exact_type == token.LPAR
True

With asttokens:

>>> asttokens.ASTTokens('()\n', True).tokens[0].type == token.OP
True
>>> asttokens.ASTTokens('()\n', True).tokens[0].exact_type == token.LPAR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Token' object has no attribute 'exact_type'

The exact type gives a more detailed type when type is generic OP value.

[alexmojaki]

What's the advantage of exact_type compared to checking the contents of the token (I can't remember the attribute - probably .string or .text)?

[hydrargyrum]

Cleaner code, clarity, using enums rather than hardcoded strings, less typo risk, easier to grep a named constant rather than a metacharacter, consistence/compatibility with code using the tokenize standard module.