Support for Non-Diacritics characters in combo boxes

[luiscla27]

Is your feature request related to a problem? Please describe.

Searching for México is quite hard. In spanish, diacritics are something that most people pays attention when they are filling "main content" but not when "searching for them", like when filling a catalog with the correct country strings, you'll find "Mexico" being written as "México", but when daily users want to select that value in a combo they'll probably write "Mexico".

This issue occurs especially with two diacritics ´ and ¨, being the most common the first one. As it's used A LOT in spanish, even people names have it, daily use objects, verbs, subjects, everything (!). For english, happens a bit because their keyboards dont have ñ but they interact with data written in spanish; So, they are unable to type "Niño" (means kid) they type "Nino" expecting to find the value.

The "México" example is the one issue I encounter the most when filling forms in the web. However, my main (real) issue are people names, some méxican examples: "José", "Óscar", "Ángel", "Germán" it gets even worse because there's names that can be written with the diacritics in two different ways, like "German" might be written without the diacritic and it's a valid name, but spelled differently.

With the component current behaviour users claim their name (or country) is missing.

Describe the solution you'd like

Filtering

I would like to forget about diacritics when searching for a value inside the igx-simple-combo or igx-combo, I want to type "Mexico" and being shown any of the following "México", "Méxícó", "mexicÓ". Diacritics shouldn't matter.

Sorting

I would like the component to automatically sort display values alphabetically considering diacritics expected order. Sorting without considering diacritics is mandatory, in spanish o and ó are the same, it's a know issue of all encodings to order them "alphabetically" with those characters really far away.

Specifically, every name starting with a diacritic character are the ones usually claimed as missing. That's because, users don't find it by using the filter, nor scrolling to its expected ordered character.

Describe alternatives you've considered

Currently I've made an implementation myself, it kind of does the following:

1. When data array is setted, Only then, and only once: I create a clone attribute called escapedText and fill it with the displayKey value
2. The escapedText attribute is filled with diacritics removed and lower-cased ("México" is saved as "mexico")
3. I use the escapedText value for two purposes, being the first one to SORT the values... this is specially needed for words starting with diacritics. Like the Óscar name. The second purpose is for filtering.
4. I implemented a pipe filter, the transform method kinda looks like this:

   /**
   * `filterTerm` must be parametrized without dielectrics 
   **/
   public transform(collection: Array<{escapedText: string}>, filterTerm: string): any[] {
   const specialCharacterRegExp = /(\+|\*|\\|\^|\.|\[|\]|\$|\&|\||\?)/gm;
   const regexp = new RegExp('^.*?' + filterTerm.replace(specialCharacterRegExp, '\\$1').replace(/\s/gi, '.*?') + '.*?$', 'is');
   return collection.filter((item) => {
     return item.escapedText.toString().match(regexp) !== null;
   });
   }

Additional context

The regexp replacing spaces \s to .*? is a plus, as I found incredibly useful to treat spaces as * wildcards (vscode does the same when using cntrl+p), I removed diacritics by using this code. Removing diacritics is a costly process, but it is not when you do it only once, being the right time when the component data is ready.

Here's an example of the issue, there's New Mexico and Old México in the available data. But by typing "Mexico", you'll find only "New Mexico"

[kdinev]

@luiscla27 I reviewed your feature request and I have one slight comment - a non-diacritic search/filter could be exposed, but it won't replace the default pipe implementation that we've had so far. That would be a breaking change, which we would prefer to avoid so we don't change the behavior of the filters in existing applications that go through an ng update. Instead, we can expose the filtering strategy for replacement, and we can provide an out-of-the-box non-diacritic strategy, but you would need to tell the component to use that strategy instead of the default one. Does this sound OK?

[luiscla27]

Yes! that'll be more than enough @kdinev, thank you so much for paying attention to this matter :)

[luiscla27]

Hello! @Lipata, I reviewed a bit the PR but I didn't saw anything about sorting, is it going to be added later or only the filter is going to be supported? thks

[rkaraivanov]

@luiscla27 Yes, we are planning to add support for sorting at a later point. This issue should not have been closed as only the filter part of it is done.

[github-actions[bot]]

There has been no recent activity and this issue has been marked inactive.

[github-actions[bot]]

There has been no recent activity and this issue has been marked inactive.

[kdinev]

@rkaraivanov @ChronosSF How large is this item in terms of sorting implementation?

[rkaraivanov]

@kdinev

I'd say close to a sprint.

The sorting pipeline is already there, it just needs to be extended for locale aware comparisons. What is more important is a good specification defining the default behavior and the API on how to control that behavior. Also, an additional PR for the web component combo element since that should be reflected there as well.

[ChronosSF]

@kdinev , we are moving this into spec + implementation for this sprint assigned to Grinders