search

GEWIS / gewisweb

GEWIS Website
https://gewis.nl
GNU General Public License v3.0
16 stars 33 forks source link

Remove `iconv` for decision search highlighting #1766

Closed tomudding closed 7 months ago

tomudding commented 7 months ago

Unfortunately, by using iconv there is a greater chance that the resulting (converted) text is either short or longer. For example, by using the euro symbol (€) we artifically increase the length of the texts we are comparing:

iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', '€');
// 'EUR'

This is problematic, as it will result in incorrectly alignments of <mark>. While this can be mitigated by carefully calculating offsets for the offsets this quickly makes it more difficult to keep maintaining this functionality. Especially when there need to be more of these exceptions.

image

Only using the transliterator with Any-Latin; Latin-ASCII seems to preserve the length of the comparing elements and allow for searching accented/special characters. There are characters that are not part of/exist in Latin-ASCII, however, these characters are probably never used in the setting of the association.

This is a bug fix for GH-1764.

tomudding commented 7 months ago

€ is not the only symbol that causes problems for us, things like special quotes ’ have the same effect.