search

champred / UPR-Android

Android UI wrapper around Universal Pokemon Game Randomizer
GNU General Public License v3.0
17 stars 0 forks source link

Special characters such as the "é" in "Pokémon" are incorrect #7

Closed fishcu closed 6 months ago

fishcu commented 1 year ago

When randomizing a HGSS ROM, the resulting ROM will have special characters replaced by the wrong characters, for example the "é" in "Pokémon", or the " ' " in "you'll".

I have done the patching with the latest released version v0.3.3 and with a clean HG ROM on an Android 12 device.

I have attached two screenshots that show the issue.

image image

fishcu commented 1 year ago

As another data point, FireRed behaves in the same fashion.

Interestingly, the wrong character mapping is the same, i.e., é => [nothing] and ' => â.

image

champred commented 1 year ago

It appears that Android replaces special characters in the source code with escape sequences. Tentatively I have hardcoded those escape sequences in the characters tables. This could potentially affect more special characters though.

champred commented 1 year ago

After doing more research it seems that é gets replaced with the bytes C3 A9 when compiling on Windows. This makes no sense considering the character byte is E9 in both UTF-8 and Windows-1252.

Similarly the apostrophe character 2019 is replaced with E2 20AC 2122 which at least makes more sense considering it's multiple bytes. It turns out that E2 is â which is why that shows up.

To fix this I set the encoding option to be UTF-8 in the Gradle build script. The unicode escape decoding might be completely unnecessary. Although strangely enough it still appears to give the correct results even with the incorrect escapes which is even more confusing.