search

Dan-Q / twee2

A Twine 2 compiler that runs from the command-line, like Twee.
https://dan-q.github.io/twee2/
GNU General Public License v2.0
79 stars 21 forks source link

Non-ASCII characters get scrambled #18

Open DizzyHSlightlyVoided opened 7 years ago

DizzyHSlightlyVoided commented 7 years ago

Unicode characters get scrambled when loaded through Twee2. For example, the back/forward arrows in the latest version of Sugarcube look like this:

sugarcubeerrorscorrect

Whereas if a story is compiled in Twee2, they look like this:

sugarcubeerrorstwee2

Also, if a file contains UTF-8 byte-order marks, Twee2 seems to interpret them as literal ASCII characters, in includes and elsewhere.

emmeken commented 7 years ago

The problem seems to be that Ruby's default encodings can be unexpected. The default encoding used for text files depends on system settings, so it might work fine for some people and be broken for others.

I think it would be good to use UTF-8 as the default encoding for text files, possibly with an override option on the command line.

A workaround would be to invoke Ruby with the --external-encoding=UTF-8 option.

mcdemarco commented 6 years ago

Some more details are mentioned in #21.

NQNStudios commented 4 years ago

This is happening to me, and mangling Spanish words in a bilingual game I'm making:

image

I see fixes and workarounds in the discussion on #21 but I feel like it should work out of the box.

I'm running twee2 on Windows 10.