search

sighingnow / libclang

(Unofficial) Release libclang (clang.cindex) on pypi.
https://pypi.org/project/libclang
Other
81 stars 21 forks source link

Support decode Token into Multi encoding type ? #60

Open Vela-zz opened 11 months ago

Vela-zz commented 11 months ago

I try to parse code that include comment with Japanese and Chinese.

when I want to get all tokens spelling in a method by iterate tokens in cursor.get_tokens(), it will raise error when it try to decode some string into UTF-8 in default.

what if I got a token and want to decode it into other format like "shiftJIS"? can add some args like "encoding" when return token.spelling ?

sighingnow commented 10 months ago

I will try submit pull request to upstream LLVM.