Closed stoey closed 10 years ago
Unicode "program"s get encoded as utf8 before being passed to tokenizer.generate_tokens and the check on expr type is relaxed from str to STR_TYPES.
Also added simple regression test that fails before the change and succeeds with it.
See #8 for rationale
Unicode "program"s get encoded as utf8 before being passed to tokenizer.generate_tokens and the check on expr type is relaxed from str to STR_TYPES.
Also added simple regression test that fails before the change and succeeds with it.