search

kevinushey / sourcetools

Tools for reading, tokenizing, and parsing R code.
MIT License
77 stars 3 forks source link

decoding of strings #9

Closed kevinushey closed 8 years ago

kevinushey commented 8 years ago

E.g. handling:

Not strictly necessary since we are more interested in the textual contents of the document rather than their interpretation.

kevinushey commented 8 years ago

Can probably use ::wcrtombs() here: http://en.cppreference.com/w/cpp/string/multibyte/wcrtomb. But there are probably subtleties in handling encoding that I don't understand yet.

kevinushey commented 8 years ago

Implemented now as stringValue() member function: https://github.com/kevinushey/sourcetools/blob/a8f0705149a381db9d9bd10e74996e7395b4a51c/inst/include/sourcetools/tokenization/Token.h#L332-L397