Bad SHIFT-JIS to UTF-8 conversions

[norikawa]

Some song names (and possibly other details such as artist names, though I've mostly only seen song names for this) are incorrect when converted to UTF-8 from SHIFT-JIS. For example, "Entropic EnĤαncemEnt" is instead converted to "Entropic En鹹αncemEnt"

For my own WIP server in Crystal, I've gone through and found most of the incorrect characters and made a list of them to automatically replace them after the conversion step:

def fix_bad_utf8(text : String)
        replacements = {
            "齷" => "é",
            "鬮" => "¡",
            "齶" => "♡", 
            "ケロH" => "ケロ⑨",
            "曦" => "à",
            "曩" => "è",
            "龕" => "€",
            "壬" => "ê",
            "驩" => "Ø",
            "=墸Σ≡=｡ﾟ:*.:+｡.☆" => "=͟͟͞ Σ≡=｡ﾟ:*.:+｡.☆",
            "鹹" => "Ĥ",
            "闃" => "Ā",
            "饌" => "²",
            "煢" => "ø",
            "餮" => "Ƶ",
            "蔕" => "ῦ",
            "盥" => "⚙︎",
            "頽" => "ä",
            "隍" => "Ü",
            "雋" => "Ǜ",
            "鬻" => "♃",
            "鬥" => "Ã",
            "鬆" => "Ý",
            "趁" => "Ǣ",
            "驫" => "ā",
            "騫" => "á",
            "齲" => "♥",
            "骭" => "ü"
        }
        replacements.each_key do |key|
            text = text.gsub(key) {replacements[key]}
        end
        return text
end

I'm not familiar enough with TypeScript to know what the equivalent functions would be, but hopefully the list of replacements can be helpful at least!

[norikawa]

I've also gone through and confirmed that the incorrect characters only appear in this particular circumstance, so this method of replacing every instance of the given character/sequence will never replace good Japanese text with bad replacement text.

I haven't confirmed that this list covers all bad conversions though, I doubt I found all of them!

[22vv0]

There is a translation table in some of the js files, so I think we'd just need to add some new characters to the list? Will check.