search

vinzenz / libpypa

libpypa is a Python parser implemented in pure C++
Apache License 2.0
189 stars 48 forks source link

Unicode support #17

Open kmod opened 9 years ago

kmod commented 9 years ago

Hey @vinzenz, we're trying to add unicode support to Pyston and it looks like pypa doesn't support unicode literals -- ex it will treat u"\u0180" as the length-6 string "\\u0180". Is this something that could get added?

Right now we're using UTF8 to store the unicode data as part of the AST, but I don't mind switching that to something else (I think our unicode implementation uses UTF-32 anyway).

undingen commented 9 years ago

This got implemented in #20. What AFAIK still missing for complete unicode support is specifying a source file encoding.