Incorrect special character sorting

realm / realm-core

Core database component for the Realm Mobile Database SDKs

https://realm.io

Apache License 2.0

1.02k stars 165 forks source link

Incorrect special character sorting #2573

Open vonox7 opened 7 years ago

vonox7 commented 7 years ago

The JVM, Swift and all other programming languages (see also unicode specification) mentioned in src/realm/unicode.cpp) define the following character sequence as "sorted: !"#$%&'()*+,-./123:;<=>?ABCXYZ_abcxyz

In contrast, Realm 3.1.2 considers the following as "sorted": '- !"#$%&()*,./:;?_+<=>123aAbBcCxXyYzZ

Code for reproduction

The realm sorting algorithm has the following bugs:

The characters ' and - are before the characters !"#
The numbers (0-9) are after all other special characters like : and =
The characters ?, + and _ are in the wrong position

The realm sorting documentation has the following missing:

Realm sort is case-insensitive (in contrast to Java/Swift greaterThan/smallerThan Operator (< / >))

It seems like realm doesn't comply to any standard (unicode, ascii...) and just sorts all special characters by a random specification and order.

kneth commented 7 years ago

@vonox7 The sorting behaviour you mention is the expected one. I have created https://github.com/realm/realm-java/issues/4527 to clarify. Moreover, I will update the documentation so it is clear tha this is the intended behaviour.

vonox7 commented 7 years ago

@kneth Thanks for the update. The documentation update explains the intended lowercase/uppercase problem. But the characters like'-!"#:=?+_ ARE in the utf-8 range (0-591). See also https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)#Table_of_characters . I think you missed here my point.

ironage commented 7 years ago

https://github.com/realm/realm-core/issues/777 is related

nalexn commented 3 years ago

I'm shocked that such divergence from all the existing specs exists for years unresolved...

How am I supposed to sort contacts by name to form a (#) group after a (z), if (z) is always the last, no matter which character I use for grouping? Have to appeal to yet another hack to make Realm work.

bmunkholm commented 3 years ago

Hey @nalexn - happy to see someone still cares! We fully understand the frustration when something like this is needed!

We surely would love to get this fixed, and while we hoped for a simple solution by adding full ICU support from the platforms, none of the mobile platforms really support that in an official way, unfortunately (despite earlier promises..). Just including all of ICU is also out of question for size reasons. So it looks like we have to get more inventive and offer some kind of opt-in solution or configuration of sorts so that we don’t blow up the library size significantly for those that don’t need this.

We can’t really promise anything about when we can take that on, but it’s not on our immediate shortlist for now. We have hundreds of good ideas in the backlog and we do try to prioritize features based on user feedback so that we give most people the highest value all the time. So definitely appreciate your "upvote" on this!

In the meantime, did you actually find a workaround? Otherwise happy to help out!

jaltin commented 2 years ago

Hi,

We also really would like to be able to control the sorting. I have outlined a suggestion in https://github.com/realm/realm-js/issues/3770, perhaps you could do something along those lines?