Closed coreyog closed 6 years ago
It's not very well documented. Here's what's going on. Here's what stantler's flavor text looked like in Gold:
The curved antlers
subtly change the
flow of air to
create a strange
space where real-
ity is distorted.
The early games stored flavor text with explicit line breaks, hyphenation, and page breaks. We represent this is the database as special characters in the text.
\xc2\xad
is the UTF-8 encoding of U+00AD)You can and should feel free to replace these characters with normal spaces or whatever. Here's what veekun does:
# Page breaks are treated just like newlines.
# Soft hyphens followed by newlines vanish.
# Letter-hyphen-newline becomes letter-hyphen, to preserve real
# hyphenation.
# Any other newline becomes a space.
html = flavor_text.replace(u'\f', u'\n') \
.replace(u'\u00ad\n', u'') \
.replace(u'\u00ad', u'') \
.replace(u' -\n', u' - ') \
.replace(u'-\n', u'-') \
.replace(u'\n', u' ')
Closing. Feel free to comment if you still have questions, or join us on IRC: https://veekun.com/chat.
So far I've found form feeds in Bulbasaur's English Pokemon Red dex entry and Stantler's English Pokemon Gold entry has an interesting control sequence I hadn't seen of 0xC2AD right where it splits the word "reality." I don't know any of the other language that exist in the data so I don't know if I should be blindly replacing these characters with spaces or just removing them. Is there a rule of thumb for how to parse these characters? Are there other character sequences I should be parsing out that I just haven't found because there's 802 pokemon in 2 dozen games, in a dozen languages?