Unicode identifiers that are not normalized

HertzDevil commented 2 years ago

The following error might or might not be expected, but it is certainly confusing to look at:

à = 1  # U+00E0
puts à # U+0061 U+0300
# Error: undefined local variable or method 'à' for top-level

This is because the compiler compares the variable names by codepoints directly and does not perform any Unicode normalization. I tested the same in other languages:

Python and Rust work;
Ruby doesn't work, even if # encoding: UTF-8 is specified;
GCC doesn't work, Clang and MSVC work. C++ will no longer accept identifiers that aren't in NFC;
Java and Zig do not allow non-ASCII characters in identifiers. Ruby does not allow them if that encoding magic comment isn't included.

straight-shoota commented 2 years ago

Just to make sure: "Python and Rust work" means they treat different non-normalized representations of the same normalized form as identical?

straight-shoota commented 2 years ago

I think I'd probably prefer to merge identifiers on their normalized form. The formatter can run normalization. That means, there would effectively be almost exclusively normalized forms in actual code (unless you don't use the formatter).

We could also consider disallowing non-normalized forms. It saves us to do normalization in the formatter. But it might be restrictive for some environments, not sure.

crystal-lang / crystal

Unicode identifiers that are not normalized #11222