Mixed Unicode normalization causing seemingly equal searches to not return results

[kisaragi-hiu]

• str.toLowerCase() === str → case insensitive search
• else case sensitive

The problem is SQLite COLLATE NOCASE only knows about ASCII characters. (We can't just downcase everything because the case matters for some words (like "AI" for instance).) We need to do an NFD normalization before saving into database.

• [x] Store strings into database in NFD so that COLLATE NOCASE works (not perfect but covers our languages) (https://github.com/kemdict/kemdict-data/commit/b312b0443bad8d76a441212d0f86f5dea33f9ed8)
• [x] Make sure strings are in NFC before displaying (25fa7db1e34d879d81794bf795c96f2de572faf7, 74c964209d6be6a254cedcbe4d1c10bce84a9082, ceab96b4bc777f117ae77b70a85a2cabaca5a3e6)
• [ ] in getHetFromUrl: do the smart case logic; do the database query accordingly; return whether smart case is on...
• [ ] ...for WordPreview, so the exact match marker can be aware of smart case

[kisaragi-hiu]

Actually, LIKE is case-insensitive by default.

The problem I faced was that when I was searching Ǹg-bāng, it didn't return any result even though it should. I thought this is an issue with casing, but after ensuring Unicode normalization it started working.

With the issue resolved, smart case is probably actually not necessary. AI and other loan words from English acronyms are probably the only cases where the word has just one correct casing, and they're unique enough to not be buried in entries with the wrong casing.

[kisaragi-hiu]

Updated the title because I keep getting confused by what has been done and what not.