search

unicode-rs / unicode-normalization

Unicode Normalization forms according to UAX#15 rules
https://unicode-rs.github.io/unicode-normalization
Other
160 stars 42 forks source link

Weird discrepancy with ICU4X #105

Open aumetra opened 1 month ago

aumetra commented 1 month ago

So, to set the scene here, I have a proptest between two libraries set up. One of the libraries uses unicode-normalization under the hood, the other icu_normalizer.

I expected that both output the same values, but my CI exploded at some point on the weird string "\u{11366}\u{113ce}". When put through the NFC normalizer, you get two different outputs:

  1. unicode-normalization: "\u{113ce}\u{11366}"
  2. icu_normalizer: "\u{11366}\u{113ce}"

Just a fun little thing I thought I'd report since it's technically a correctness issue (I'm just not good enough with Unicode to determine whether it's an issue with ICU4X or this crate).

aumetra commented 1 month ago

One more. I don't know why proptest suddenly finds so many:

Original: "\u{113c2}\u{113b8}"
unicode-normalization: "\u{113c7}"
icu_normalizer: "\u{113c2}\u{113b8}"
Manishearth commented 1 month ago

This crate hasn't been updated to Unicode 16.0 yet. Doing so is not super straightforward this time due to some of the newer characters having interesting combinations of properties.

aumetra commented 1 month ago

Ah, interesting. Thanks for the info! Good to know this is due to a new standard revision and not due to a bug